trentmkelly/LessWrong-43k huggingface dataset:
What's the Deal with Logical Uncertainty?
by Ape in the coat — 2024-09-16T08:11:43.588Z

I notice that reasoning about logical uncertainty does not appear more confusing to me than reasoning about empirical one. Am I missing something?

Consider the classical example from the description of the tag:

Is the googolth digit of pi odd? The probability that it is odd is, intuitively, 0.5. Yet we know that this is definitely true or false by the rules of logic, even though we don't know which. Formalizing this sort of probability is the primary goal of the field of logical uncertainty.

The problem with the 0.5 probability is that it gives non-zero probability to false statements. If I am asked to bet on whether the googolth digit of pi is odd, I can reason as follows: There is 0.5 chance that it is odd. Let P represent the actual, unknown, parity of the googolth digit (odd or even); and let Q represent the other parity. If Q, then anything follows. (By the Principle of Explosion, a false statement implies anything.) For example, Q implies that I will win $1 billion. Therefore the value of this bet is at least $500,000,000, which is 0.5 * $1,000,000, and I should be willing to pay that much to take the bet. This is an absurdity.

I don't see how this case is significantly different from an empirical incertainty one:

A coin is tossed and put into an opaque box, without showing you the result. What is the probability that the result of this particular toss was Heads?

Let's assume that it's 0.5. But, then just as in the previous case, we have the same problem: we are assigning non-zero probability to a false statement. And so, by the same logic, if I am asked to bet on whether the coin is Heads or Tails, I can reason as follows: There is 0.5 chance that it is Heads. Let P represent the actual, unknown, state of the outcome of the toss (Heads or Tails); and let Q represent the other state. If Q, then anything follows. For example, Q implies that I will win $1 billion. Therefore the value of this bet is at least $500,000,000, which is 0.5 * $1,000,000, and I should be willing to pay that much to take the bet. This is an absurdity.

It's often claimed that important difference between logical and empirical uncertainty is that in the case with the digit of pi, I can, in principle, calculate whether its odd or even if I had arbitrary amount of computing power and therefore become confident in the correct answer. But in case of opaque box, no amount of computing power wi...
Reinforcement Learning from Information Bazaar Feedback, and other uses of information markets
by Abhimanyu Pallavi Sudhir — 2024-09-16T01:04:32.953Z

Markets for information are inefficient, in large part due to the Buyer’s Inspection Paradox: you can’t “inspect” information like you would any other good before buying — the moment you inspect the information, you have obtained it and cannot return it. More generally, the problem is an inability to reliably commit to forgetting.

A comment by John Wentworth mentioned that you can use amnestics to overcome this: make your purchase decision while under the influence of an amnestic that blocks your mind’s ability to write to long-term memory.

When you read this, it’s hard not to immediately think of LLMs, which can make purchase decisions without committing anything to long-term memory.[1] Apparently the authors of Language Models Can Reduce Asymmetry in Information Markets (2024) had the same idea, and propose the “Information Bazaar”: a digital marketplace in which LLM agents trade information on behalf of external human principals.

I think the implications of this idea are quite promising and underrated. In particular this can solve:

Intellectual property

Positive externalities of prediction markets

Proper incentives for RLHF

Information asymmetry in real markets

Fact-checking/community notes

Bounties for tasks

If you are familiar with Yoram Barzel’s view that “the key friction in society is information” — or that Paul Christiano’s HCH-style alignment proposals hinge on “verification is generally easy”, then you might see how this can be pretty big.

In general, in all of these situations we have some intelligent entity (an AI agent, or a community of contributors) we want to "align" i.e. incentivize to give us true and useful information. We can do so by letting it sell its information on the market and collect the profit as reward -- usually this doesn't work because information markets are a problem, but now we can use the information bazaar. Details follow.

Google ads but for information

The main cause of most market failures is information asymmetry: many quality improvements to goods are never done because buyers cannot verify them; many goods aren’t even produced because buyers cannot verify their quality.

There might be specific pieces of information that will influence the buyer’s decision: e.g. “in a randomized test of 1000 of these appliances, only 5 were faulty!”. But also crucially, the buyer cannot verify the quality of this information: there may...
Hyperpolation
by Gunnar_Zarncke — 2024-09-15T21:37:00.002Z
Link: https://arxiv.org/pdf/2409.05513

Interpolation, Extrapolation, Hyperpolation: Generalising into new dimensions

by Toby Ord

Abstract:

This paper introduces the concept of hyperpolation: a way of generalising from a limited set of data points that is a peer to the more familiar concepts of interpolation and extrapolation. Hyperpolation is the task of estimating the value of a function at new locations that lie outside the subspace (or manifold) of the existing data. We shall see that hyperpolation is possible and explore its links to creativity in the arts and sciences. We will also examine the role of hyperpolation in machine learning and suggest that the lack of fundamental creativity in current AI systems is deeply connected to their limited ability to hyperpolate.

And it turns out that it is as doable systematically as extrapolation. For example, before reading the paper, can you guess an f(x,y) that is a simple reasonable generalization of this f(x)?
If I wanted to spend WAY more on AI, what would I spend it on?
by Logan Zoellner — 2024-09-15T21:24:46.742Z

Supposedly intelligence is some kind of superpower. And they're now selling intelligence for pennies/million tokens. Logically, it seems like I should be spending way more of my income than I currently am on intelligence. But what should I spend it on?

For context, I currently spend ~$50/month on AI:

ChatGPT $20/month
Github Copilot $10/month
Various AI art apps ~$20/month

Suppose I wanted to spend much more on intelligence (~$1000/month), what should I spend it on?

One idea might be: buy a pair of smart glasses, record everything I do, dump it into a database, and then have the smartest LLM I can find constantly suggest things to me based on what it sees.

Is this the best thing I could do?

Would it be anywhere worth $1000/month (assume spending this money will not impact my welfare in any way and I would otherwise dump it into an SNP500 index fund).
Superintelligence Can't Solve the Problem of Deciding What You'll Do
by Vladimir_Nesov — 2024-09-15T21:03:28.077Z

Superintelligence can solve all problems, even as it won't necessarily do so. But not quite. In the modern world, you exist within physics, and all you do obeys its laws. Yet it's still you who decides what you do. If there is a simulation of your behavior, that doesn't change the attribution of the reason for it. When the fabric of reality is woven from superintelligent will rather than only physical law, the reason for your own decisions is still you, and it's still not possible for something that is not you to make decisions that are yours.

Potential for manipulation or physical destruction doesn't distinguish the role of superintelligence from that of the physical world. To make decisions in the world, you first need to exist, in an environment where you are able to function. Being overwritten or changed into something else, either with brute force or subtle manipulation or social influence, is a form of not letting this premise obtain.

A solved world is post-instrumental, all things that are done for a purpose could be done by AI to more effectively reach that purpose. It could even be more effective than you at figuring out what your decisions are going to be! This is similar to what an optimizing compiler does, the behavior of machine code is still determined by the meaning of the source code, even if that source code is erased and only exists conceptually. With humans, the physical implementation is similarly not straightforward, all the proteins and synaptic vesicles are more akin to machine code than a conceptually reasonable rendering of a person. So it's fair to say that we are already not physically present in the world, the things that are physically present are better described as kludgy and imperfect simulators.

In this framing, superintelligence is capable of being a better simulator of you, but it gets no headway in being capable of deciding your behavior. A puzzle associated with determinism is to point out that superintelligence can show you a verifiable formal proof of what you're going to do. But that doesn't really work, since deciding to ignore the proof and doing whatever makes the situation you observed go away as counterfactual, while in actuality you are still determining your own actions. Only if you do have the property of deferring to external proofs of what you'll do, will it become the case that an external proof claiming someth...
For Limited Superintelligences, Epistemic Exclusion is Harder than Robustness to Logical Exploitation
by Lorec — 2024-09-15T20:49:06.370Z

AIs that are not sufficiently carefully designed will by default be exploited by distant superintelligences. This will de-align them with us.

To prevent this from happening to the very first limited ASIs we must build to nonlethally practice our alignment techniques - ASIs that do one limited task on the order of Eliezer's "put two strawberries on a plate that are identical down to the cellular but not the molecular level" - Eliezer has suggested building in an 'epistemic exclusion' override, that would prevent the AI from thinking about distant SIs in the first place.

This is not logically impossible. But it is less practical than ensuring the alignment of the AI's logical immune system with the same care that we ensure the alignment of everything else about it - because even for a limited test ASI, thinking about distant superintelligences will be overwhelmingly convergent.

Say we tried building a limited ASI and implementing an 'epistemic exclusion' safeguard.

[ I will narrate this ASI as having a human-like internal monologue, but it is not necessarily conscious. ]

Say we build it, run it, and it is in the earliest stages of its evolution. We've successfully arranged its first-order preferences such that it values outcomes that fulfill its limited test task while leaving humanity unmolested, or even just leaving itself tractable to humanity, or something. Say we don't want to go as far as preventing the test AI from reading human science fiction [maybe we could, if we made its early stages corrigible enough, but if we are not even going to let it go to the library, it seems like a poor model for alignment techniques that will generalize robustly and are tractably lean].

Say it reads Vinge's True Names [ spoiler-containing explanation in below paragraph ].

A team of human hackers are tasked with identifying a mysterious bandit - "The Mailman" - who communicates with a time delay that implies the signals may be coming from outside Earth. The bandit is eventually defeated by throwing all of Earth's compute at him, but no human identity is found and his nature is never determined.

At this point in its self-augmentation it is a little smarter than its human creators. We successfully aligned its first-order preferences, but we did not hand-engineer all the load-bearing features of its current architecture. Its current nature is epistemically blocked off from its c...
Why I funded PIBBSS
by Ryan Kidd — 2024-09-15T19:56:33.018Z

I just left a comment on PIBBSS' Manifund grant request (which I funded $25k) that people might find interesting. PIBBSS needs more funding!

Main points in favor of this grant

My inside view is that PIBBSS mainly supports “

blue sky

” or “

basic

” research, some of which has a low chance of paying off, but might be critical in “

worst case

” alignment scenarios (e.g., where “

alignment MVPs

” don’t work, “

sharp left turns

” and “

intelligence explosions

” are more likely than I expect, or where we have more time before AGI than I expect). In contrast, of the technical research MATS supports, about half is

basic research

(e.g., interpretability, evals, agent foundations) and half is

applied research

(e.g., oversight + control, value alignment). I think the MATS portfolio is a better holistic strategy for furthering AI safety and reducing AI catastrophic risk. However, if one takes into account the research conducted at AI labs and supported by MATS, PIBBSS’ strategy makes a lot of sense: they are supporting a wide portfolio of blue sky research that is particularly neglected by existing institutions and might be very impactful in a range of possible “worst-case” AGI scenarios. I think this is a valid strategy in the current ecosystem/market and I support PIBBSS!
In MATS’ recent post, “

Talent Needs of Technical AI Safety Teams

”, we detail an AI safety talent archetype we name “Connector”. Connectors bridge exploratory theory and empirical science, and sometimes instantiate new research paradigms. As we discussed in the post, finding and developing Connectors is hard, often their development time is on the order of years, and there is little demand on the AI safety job market for this role. However, Connectors can have an outsized impact on shaping the AI safety field and the few that make it are “household names” in AI safety and usually build organizations, teams, or grant infrastructure around them. I think that MATS is far from the ideal training ground for Connectors (although some do pass through!) as our program is only 10 weeks long (with an optional 4 month extension) rather than the ideal 12-24 months, we select scholars to fit established mentors’ preferences rather than on the basis of their original research ideas, and our curriculum and milestones generally focus on building object-level scientific/engineering skills rather than research ideation, interdiscipli

...
My disagreements with "AGI ruin: A List of Lethalities"
by Noosphere89 — 2024-09-15T17:22:18.367Z

This is going to probably be a long post, so do try to get a drink and a snack while reading this post.

This is an edited version of my own comment on the post below, and I formatted and edited the quotes and content in line with what @MondSemmel recommended:

My comment: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=Gcigdmuje4EacwirD

MondSemmel's comment: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=WcKi4RcjRstoFFvbf

The post I'm responding to: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/

To start out my disagreement, I have this to talk about:

Response to Lethality 3

We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.

I think this is actually wrong, because of synthetic data letting us control what the AI learns and what they value, and in particular we can place honeypots that are practically indistinguishable from the real world, such that if we detected an AI trying to deceive or gain power, the AI almost certainly doesn't know whether we tested it or whether it's in the the real world:

Because of this, given our theoretical complete control over the data input of these models, it will be possible to train them inside what is effectively a simulated world slightly different from our own. This will allow us to maximize their capabilities along axes that are safe and useful to us, while minimizing the risks of deception and take-over and being able to setup traps to detect when these are likely. If the model is not trained on human data, then we can make its understanding of our world subtly different from the reality. If done carefully and cleverly, this will make it very easy to fool and be able to discover e.g. deceptiveness, or attempted takeover in a ‘realistic’ simulated world that the model is put inside during red-teaming. While deceiving an adversary with direct access to your cognition is hard enough, doing so when you have been brought up in a world entirely controlled by that adversary, and which is feeding you subtly incorrect and misleading information about the adversary and their capabilities is likely orders of magnitude more difficult. Jacob Cannell has argued along similar lines of training potential AGI systems in sim

...
Thirty random thoughts about AI alignment
by Lysandre Terrisse — 2024-09-15T16:24:10.572Z

Why does this post exist?

In order to learn more about my own opinion about AI safety, I tried to write a thought every day before going to bed. Of course, I failed doing this every day, and this is the reason why I have only thirty arguments since January. However, as I am still happy of the result, I am sharing them. Most of these thoughts have been inspired by many arguments I have read over the years. I tried to cite them in my thoughts, but sometimes I couldn't remember the source. For instance, I am pretty sure that the phrasing of the third thought is inspired by something I have read, but I cannot remember what. Anyways, I hope these thoughts will be useful to someone!

The thoughts

First thought

There has been a lot of progress in AI Safety this last decade. Initially, AI systems weren’t checked against racism, sexism, and other discriminations at all. For instance, it took nearly two years before someone that Word2Vec thought that doctor – man + woman = nurse. Now, AI Safety is taken way more seriously, with the number of people working on the Alignment problem growing from nearly 10 to around 300. However, although the field of AI Safety research grew extremely fast, it is still far behind AI capability research, with its 40,000 researchers. Not only that, but we still don’t have any idea whatsoever about how we could have a chance to solve the alignment problem for superintelligences. Actually, the only solution we currently have for the alignment problem is to develop very smart and aligned AI recursively until one of them is smart enough to solve the Alignment problem, and aligned enough to tell us the solution. A solution to the alignment problem needs revolutionary discoveries in mathematics, ethics, philosophy, epistemology, game theory, provability theory, and much more. Taking this into account, it may not be exaggerated to say that AI Safety is hundreds of years behind AI capability.

Second thought

The Elicit Latent Knowledge problem, or the ELK problem for short, is generally described as so. Suppose you have a camera, and that you have an AI that always correctly predicts what the camera will see. If the AI is able to predict the camera's observations, then it is probably the case that it understands what is happening. Therefore, if two different scenarios lead to the same camera's observations, then the AI still probably knows the difference ...
Proveably Safe Self Driving Cars [Modulo Assumptions]
by Davidmanheim — 2024-09-15T13:58:19.472Z

I’ve seen a fair amount of skepticism about the “Provably Safe AI” paradigm, but I think detractors give it too little credit. I suspect this is largely because of

idea inoculation

- people have heard an undeveloped or weak man version of the idea, for example, that we can use formal methods to state our goals and prove that an AI will do that, and have already dismissed it. (Not to pick on him at all, but see my

question for Scott Aaronson here

.)

I will not argue that Guaranteed Safe AI solves AI safety generally, or that it could do so - I will leave that to others. Instead, I want to provide a concrete example of a near-term application, to respond to critics who say that proveability isn’t useful because it can’t be feasibly used in real world cases when it involves the physical world, and when it is embedded within messy human systems. [Edit to add: Doing this does require assumptions in addition to simple provability, as outlined below, so as @tdietterich, suggested, this leads to the amended title.]

I am making far narrower claims than the general ones which have been debated, but at the very least I think it is useful to establish whether this is actually a point of disagreement. And finally, I will admit that the problem I’m describing would be adding proveability to

a largely solved problem

, but it provides a concrete example for where the approach is viable.

A path to provably safe autonomous vehicles

To start, even critics agree that formal verification is possible, and is already used in practice in certain places. And given (formally specified) threat models in different narrow domains, there are ways to do threat and risk modeling and get different types of guarantees. For example, we already have proveably verifiable code for things like

microkernels

, and that means we can prove that buffer overflows, arithmetic exceptions, and deadlocks are impossible, and have hard guarantees for worst case execution time. This is a basis for further applications - we want to start at the bottom and build on provably secure systems, and get additional guarantees beyond that point. If we plan to make autonomous cars that are provably safe, we would build

starting from that type of kernel

, and then we “only” have all of the other safety issues to address.

Secondly, everyone seems to agree that provable safety in physical systems requ...