trentmkelly/LessWrong-43k huggingface dataset:
An anti-inductive sequence
by Viliam — 2024-08-14T12:28:54.226Z

I was thinking about what would it mean for a sequence of bits to be "anti-inductive". It probably is a concept that is already known (as a rule of thumb, if I can think about it, someone probably already wrote a paper on it 50 years ago), but I haven't heard about it.

*

Some sequences are predictable and can be compressed. These two concepts are deeply related, because if you can successfully predict the next part of the sequence, you don't need to actually write it down; hence compression. A completely random sequence of bits cannot be compressed or predicted.

There is a simple mathematical proof that some sequences cannot be compressed, although it doesn't say which ones. For any natural number N, there are more sequences of size exactly N, than sequences of size smaller than N. Therefore no program can generate a unique sequence shorter than N for any input sequence of size N.

*

Things get more complicated if we consider the caveat that although random sequences in general cannot be compressed, true randomness means that sometimes we accidentally get a sequence that can be compressed -- for example, with probability 1/2ᴺ we get a sequence of N zeroes, and it would sound silly to argue that we can't compress that!

The solution to this paradox is that if we decide to compress only some selected sequences, then we need to add an extra bit of information specifying whether this sequence was compressed or not. Otherwise, if we see a sequence of bits saying (in binary) "a sequence of thousand zeroes", we wouldn't know whether the intended value is this very sequence of bits taken literally, or the sequence of thousand zeroes. One bit doesn't seem like much, but actually most sequences cannot be compressed, so the cost of adding one extra bit to each of them outweighs the occasional space we would save by compressing the ones we can.

But still, if I needed a random sequence of bits to use e.g. as a password for something important... and by a miracle I generated a sequence of N zeroes... I would probably feel quite uncomfortable to use it, and would simply generate a new one. Is this a good security practice, or not? Because from certain perspective, by removing the "insufficiently random" sequences from the pool of possible choices, I am reducing the size of the pool, which... kinda makes it easier to guess the password?

Something similar actually happened to me once....
Rabin's Paradox
by Charlie Steiner — 2024-08-14T05:40:25.572Z

Quick psychology experiment

Right now, if I offered you a bet[1] that was a fair coin flip, on tails you give me $100, heads I give you $110, would you take it?

Got an answer? Good.

Hover over the spoiler to see what other people think:

About 90% of undergrads will reject this bet[2].

Second part now, if I offered you a bet that was a fair coin flip, on tails you give me $1000, on heads I give you $1,000,000,000, would you take it?

Got an answer?

Hover over the spoiler to reveal Rabin's paradox[3]:

If you rejected the first bet and accepted the second bet, just that[4] is enough to rule you out from having any[5] utility function consistent with your decisions.[6]

What? How?

The general sketch is to suppose there was some utility function that you could have (with the requisite nice properties), and show that if you reject the first bet (and would keep rejecting it within a couple-thousand-dollar domain), you must have an extreme discount rate when the betting amounts are extrapolated out.

If you reject the first bet, then the average utility (hypothesizing some utility function U) of engaging in the bet is less than the status quo: 0.5⋅U(W+$110) + 0.5⋅U(W-$100) < U(W). In other words, the positive dollars are worth, on average, less than 10/11 as much as the negative dollars.

But if you keep rejecting this bet over a broad range of possible starting wealth W, then over every +$110/-$100 interval in that range the positive dollars are worth less than 10/11 the negative dollars. If every time you move up an interval you lose a constant fraction of value, that's exponential discounting.

How to turn this into a numerical answer? Well, just do calculus on the guess that each marginal dollar is worth exponentially less than the last on average.

f≡e−kw

1200∫0−100f+1220∫1100f=f(0)

1200k(e100k−1)+1220k(1−e−110k)=1

The step of the calculation where you plug it into wolframalpha

k≈0.0013

Some numbers, given this model:

The benefit from gaining the first $1 is about 1 utilon.

The maximum benefit from gaining $1,000,000,000 is a mere seven-hundred and sixty nine utilons. This is also the modeled benefit from gaining infinity dollars, because exponential discounting.

The minimum detriment from losing $1000 is over two thousand utilons.

So a discount of 100/110 over a span of $210 seems to imply that there is no amount of positive money you should accept against a loss of...
Announcing the $200k EA Community Choice
by Austin Chen — 2024-08-14T00:39:37.350Z
Link: https://manifund.substack.com/p/announcing-the-200k-ea-community

Manifund is hosting a $200k funding round for EA community projects, where the grant decisions are made by you. You can direct $100-$800 of funding towards the projects that have helped with your personal journey as an EA. Your choices then decide how $100k in matching will be allocated, via quadratic funding!

Sign up

here

to get notified when the projects are live, or read on to learn more!

Timeline

Phase 1: Project registrations open (Aug 13)

Project organizers can now create a page for their projects

here

, to raise funding as part of this EA Community Choice round.
Community members can sign up for updates

here

, or recommend projects to sign up

here

.

Phase 2: Community members receive funds (Aug 20)

We’ll give everyone $100 to donate; more if you’ve been active in the EA community. Fill out a 2-minute form to claim your $100, plus bonuses for:

Donor: $100 for taking the GWWC 🔸10% Pledge
Organizer: $100 for organizing any EA group
Scholar: $100 for having 100 or more karma on the EA Forum
Volunteer: $100 for volunteering at an EAG(x), Future Forum, or Manifest
Worker: $100 for working fulltime at an EA org, or full-time on an EA grant
Senior: $100 for having done any of the above prior to 2022
Insider: $100 if you had a Manifund account before August 2024

You can then donate your money to any project in the Community Choice round!
You can also leave comments about any specific project. This is a great way to share your experiences with the project organizer, or the rest of the EA community.
Funds in Phase 2 will be capped at $100k, first-come-first-served.

Phase 3: Funds matched and sent to projects (Sep 1)

Projects and donations will be locked in at the end of August. Then, all money donated will be matched against a $100k quadratic funding pool.
Unlike a standard 1:1 match, quadratic funding rewards a project with lots of small donors more than a project with few big donors. The broader the support, the bigger the match!

Specifically, the match is proportional the square of the sum of square roots of individual donations. A toy example:

Learn more about principles behind quadratic funding by reading

Vitalik Buterin’s explainer

, watching

this video

, or

playing with this simulator

.

What is an EA community project?

We don’t have a strict definition, but roughly any project which helps you: learn about EA, connect with the EA community, or grow your impact in the world. We’re cast...
Debate: Is it ethical to work at AI capabilities companies?
by Ben Pace — 2024-08-14T00:18:38.846Z

Epistemic status: Soldier mindset. These are not (necessarily) our actual positions, these are positions we were randomly assigned, and for which we searched for the strongest arguments we could find, over the course of ~1 hr 45 mins.

Sides: Ben was assigned to argue that it's ethical to work for an AI capabilities company, and Lawrence was assigned to argue that it isn't.

Reading Order:

Ben and Lawrence drafted each round of statements

simultaneously.

This means that each of Lawrence statements you read were written without Lawrence having read Ben's statements that are immediately proceeding.

Ben's Opening Statement

Ben Pace

I think it is sometimes ethical to work for AI capabilities companies.

I think the standard argument against my position is:

It is bad to cause an existential failure for humanity.
For these companies, we have a direct mechanism by which that may happen.
Therefore, you should not work for them.

(I am granting that we both believe they are risking either extinction or takeover by an ~eternal alien dictatorship.)

I think this argument is strong, but I argue that there are some exceptions.

I think that there are still four classes of reason for joining.

First, you have a concrete mechanism by which your work there may prevent the existential threat. For instance, there were people in the Manhattan project improving the safety of the atom bomb engineering, and I think they had solid mechanistic reasons to think that they could substantially improve the safety. I’m not sure what probability to assign, because I think probabilities-without-mechanisms are far more suspect and liable to be meaningless, but if you have a concrete story for averting the threat, I think even a 10% chance is sufficient.

Second, I have often felt it is important to be able to join somewhat corrupt institutions, and mitigate the damage. I sometimes jokingly encourage my friends at IMO corrupt institutions to rise through the ranks and then dramatically quit when something unusually corrupt happens, and help start the fire for reform. I recall mentioning this to Daniel Kokotajlo who’s resignation was ultimately very influential (to be clear I don't believe my comment had any impact on what happened). That said, I think most random well-intentioned people should not consider themselves strong enough to withstand the forces of corruption that will assault them on the inside of such

...
Fields that I reference when thinking about AI takeover prevention
by Buck — 2024-08-13T23:08:54.950Z
Link: https://redwoodresearch.substack.com/p/fields-that-i-reference-when-thinking

Is AI takeover like a nuclear meltdown? A coup? A plane crash?

My day job is thinking about safety measures that aim to reduce catastrophic risks from AI (especially risks from egregious misalignment). The two main themes of this work are the design of such measures (what’s the space of techniques we might expect to be affordable and effective) and their evaluation (how do we decide which safety measures to implement, and whether a set of measures is sufficiently robust). I focus especially on AI control, where we assume our models are trying to subvert our safety measures and aspire to find measures that are robust anyway.

Like other AI safety researchers, I often draw inspiration from other fields that contain potential analogies. Here are some of those fields, my opinions on their strengths and weaknesses as analogies, and some of my favorite resources on them.

Robustness to insider threats

The single field that seems most closely analogous to AI control is reducing vulnerability to threats resulting from insiders, especially threats that arise from access to IT systems. That is, you have a bunch of employees who need to have access to various IT systems in order to do their jobs, and you’re worried that some of these employees will abuse their access in order to accomplish some nefarious purpose. In a bank, you might worry about your tellers withdrawing money from customer accounts inappropriately. In a software company, you might worry about your employees sabotaging your code out of spite, violating user privacy for fun, or stealing your data to give it to a competitor.

This is quite similar to the problem faced by AI developers when they’re trying to control AIs (especially in internal deployments, which I argue is important here):

All of the access the AI has is access the developer decided to give it; the developer faces a constant tradeoff between the productivity of the AI (which is maximized by being permissive) and the danger it poses (which is minimized by being restrictive). And obviously, because it’s an AI it only has access to resources that are connected to the computer network where it’s running, so we can restrict our attention to the IT resources.
It’s easy for the developer to constantly surveil the AI, which substantially reduces the space of strategies the AI can take.
The developer has a lot of control over workflows, and is able to choose

...
Ten counter-arguments that AI is (not) an existential risk (for now)
by kwiat.dev — 2024-08-13T22:35:15.341Z

This is a polemic to the ten arguments post. I'm not a regular LW poster, but I'm an AI researcher and mild-AI-worrier.

I believe that AI progress, and the risks associated with it, is one of the most important things to figure out as humanity in the current year. And yet, in most discussions about x-risk, I find myself unaligned with either side.

My overall thesis about AI x-risk is that it's absolutely real, but also far enough into the future that at this moment, we should simply continue progress on both capabilities and safety. I'm not trying to argue that sufficiently powerful AI could never pose an x-risk, this belief seems rather silly.

Disclaimers:

This is largely thinking out loud, describing why I personally disagree (or agree) with the listed arguments. In the best case scenario, maybe I'll convince someone, or someone will convince me - I'd hate to be wrong on this and actively contribute to our doom.
In some cases, I could go a few steps ahead in the discussion, provide a rebuttal to my arguments and a re-rebuttal to that. I'm consciously not doing that for several reasons which are hopefully intuitive and not important enough to list here.
I'm deliberately only commenting on the summaries, and not the entire body of work behind each summary, mainly to keep things tractable. If some piece of text convinces humanity about the seriousness of x-risk, it won't be an enormous Harry Potter fanfiction (no offense). I like brevity.

Competent non-aligned agents

Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals
Humans won’t figure out how to make systems with goals that are compatible with human welfare and realizing human values
Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals
Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values

I agree that we will build agents - we already try to do that.
We only need them to be as aligned as they are powerful. A chatbot that doesn't understand actions or function calling won't be an x-risk, no matter how violently misaligned it is.
It does seem natural that we will favor competent systems. We will also favor aligned systems, at surface-aligned - in fact, the usefulness of a system is directly correlated with both the capabi

...
Alignment from equivariance
by hamishtodd1 — 2024-08-13T21:09:11.849Z

Epistemic status: research direction

Equivariance is a way of enforcing a user-specified sense of consistency across inputs for a NN. It's a concept from computer vision which I think could be used to align AIs, or rather for building an interface/framework for aligning them.

So far it has been applied exclusively in CV - so it's speculation (by me) that you could apply it beyond CV. But I think I can convince you it's a reasonable speculation because, as you'll see, equivariance and consistency, for neural networks, are deeply linked mathematically.

Like me I imagine you expect (/hope!) that "consistency with the law" or "moral consistency" will turn out to be formalizable-enough to be part of the design of a computer programs. And I could almost go as far as to say that those vague senses of the word "consistency", when you formalize them, simply are equivariance. So it makes sense to me that a reasonable design program aiming towards moral or legal consistency could be started by trying to generalize equivariance.

Equivariance as we currently have it

Here's the abstract picture you should, and usually do, see on the first page of any paper about equivariance - don't worry if you don't get it based on this:

Let's see examples - here's one from equivariance king Maurice Weiler:

If I'm right, you who read alignment papers will be seeing lots more of these diagrams in future, so you might as well get used to them! Here are two more from Divyanshu Mishra and Mark Lukacs:

These are called commutative diagrams, and a neural network is equivariant to X if for all inputs that differ by X, its outputs differ by X'. On the left there is translation equivariance, on the right is scale equivariance.

Do you

always

want translation equivariance, or scale equivariance in computer vision? No. If you were training one on satellite photography that was all guaranteed to be at the same scale, you (probably) wouldn't want scale equivariance - otherwise your NN might not be able to distinguish a real truck from a toy truck left in someone's lawn. Or, if you were training a NN on mammograms that were completely guaranteed to be aligned a certain way and centered on a certain point, you might find translation equivariance to be a bad thing too.

But, often equivariance can be really useful. It's a guarantee you can offer a customer, an example of "debugability" - hard to come by in d...
[LDSL#6] When is quantification needed, and when is it hard?
by tailcalled — 2024-08-13T20:39:45.481Z

This post is also available on my Substack.

In the previous post, I discussed the possibility of doing ordinal comparison between different entities. Yet usually when we think about measurement, we think of it as involving putting numbers to things, not just ranking them.

There are quite a few methods to convert rankings into numbers, such as Elo scores or item response theory. Different such methods generally yield extremely similar results, and are often based probabilities. For instance, in Chess, if two players’ ability differ by 400 Elo points, then the odds of the better player winning are 10:1.

Are these methods reasonable? Can they go wrong?

Quantified ordinals are ~log scales

If you look at the formula for Elo scores, it involves an exponential function, so each time the Elo changes by 400, the odds of winning changes by a factor of 10:1 (by definition). A similar principle applies to the probability of solving a difficult task as a function of IQ, if one uses item response theory.

Another way to see it is to look at the practical outcomes. For instance, IQ appears to be exponentially related to income.

But I think the most fundamental way to understand it is that these methods assume that measurement error is independent of the measured quantity, such that there is the same amount of error in the ranking of the best as in the ranking of the worst. In linear scales, measurement error is usually proportional to the measured quantity, so in order for it to be independent, one must take the logarithm.

Log scales need a base for addition

It might be tempting to think that a log scale is equivalent to a linear scale, since you can just take the exponential of it. However, there are many different exponential functions: 2^x, e^x, 10^x, ….

If n=a^x, and m=a^y, then nm=a^(x+y). So we can add log-scaled numbers, and this simply corresponds to multiplying their linearly-scaled values, even if we don’t know what base is most natural.

However, there is no corresponding expression for n+m. We can at best give some limiting values, e.g. as (y-x)ln(a) goes to infinity, n+m approaches a^y. We can use this limit to approximate n+m as a^max(x, y), but this approximation fails badly when you start summing up tons of values of similar size. (In practical terms, a tank plus a speck of dust is equal to a tank, but an apple and an orange is more than just an apple or just an orange....
A computational complexity argument for many worlds
by jessicata — 2024-08-13T19:35:10.116Z
Link: https://unstableontology.com/2024/08/13/a-complexity-theoretic-argument-for-many-worlds/

The following is an argument for a weak form of the many-worlds hypothesis. The weak form I mean is that there are many observers in different branches of the wave function. The other branches "actually exist" for anthropic purposes; some observers are observing them. I've written before about difficulties with deriving discrete branches and observers from the Schrödinger equation; I'm ignoring this difficulty for now, instead assuming the existence of a many-worlds theory that specifies discrete branches and observers somehow.

To be clear, I'm not confident in the conclusion; it rests on some assumptions. In general, physics theories throughout history have not been completely correct. It would not surprise me if a superintelligence would consider many-worlds to be a false theory. Rather, I am drawing implications from currently largely accepted physics and computational complexity theory, and plausible anthropic assumptions.

First assumption: P != BQP. That is, there are some decision problems that cannot be decided in polynomial time by a classical computer but can be decided in polynomial time by an idealized quantum computer. This is generally accepted (RSA security depends on it) but not proven. This leaves open the possibility that the classically hardest BQP problems are only slightly harder than polynomial time. Currently, it is known that factorizing a b-bit integer can be done in roughly O(exp(cb1/3)) time where c is a constant greater than 1, while it can be done in polynomial time on an idealized quantum computer. I want to make an assumption that there are decision problems in BQP whose running time is "fast-growing", and I would consider O(exp(cb1/3)) "fast-growing" in this context despite not being truly exponential time. For example, a billion-bit number would require at least exp(1000) time to factorize with known classical methods, which is a sufficiently huge number for the purposes of this post.

Second assumption: The universe supports BQP computation in polynomial physical resources and clock time. That is, it's actually possible to build a quantum computer and solve BQP problems in polynomial clock time with polynomial physical resources (space, matter, energy, etc). This is implied by currently accepted quantum theories (up to a reasonably high limit of how big a quantum computer can be).

Third assumption: A "computational density anthrop...