trentmkelly/LessWrong-43k huggingface dataset:
A Proof of Löb's Theorem using Computability Theory
by jessicata — 2023-08-16T18:57:41.048Z
Link: https://unstableontology.com/2023/08/16/a-proof-of-lobs-theorem-using-computability-theory/

Löb’s Theorem states that, if PA⊢□PA(P)→P, then PA⊢P. To explain the symbols here:

PA is Peano arithmetic, a first-order logic system that can state things about the natural numbers.

PA⊢A means there is a proof of the statement A in Peano arithmetic.

□PA(P) is a Peano arithmetic statement saying that P is provable in Peano arithmetic.

I'm not going to discuss the significance of Löb's theorem, since it has been discussed elsewhere; rather, I will prove it in a way that I find simpler and more intuitive than other available proofs.

Translating Löb's theorem to be more like Gödel's second incompleteness theorem

First, let's compare Löb's theorem to Gödel's second incompleteness theorem. This theorem states that, if PA⊢¬□PA(⊥), then PA⊢⊥, where ⊥ is a PA statement that is trivially false (such as A∧¬A), and from which anything can be proven. A system is called inconsistent if it proves ⊥; this theorem can be re-stated as saying that if PA proves its own consistency, it is inconsistent.

We can re-write Löb's theorem to look like Gödel's second incompleteness theorem as: if PA+¬P⊢¬□PA+¬P(⊥), then PA+¬P⊢⊥. Here, PA+¬P is PA with an additional axiom that ¬P, and □PA+¬P expresses provability in this system. First I'll argue that this re-statement is equivalent to the original Löb's theorem statement.

Observe that PA⊢P if and only if PA+¬P⊢⊥; to go from the first to the second, we derive a contradiction from P and ¬P, and to go from the second to the first, we use the law of excluded middle in PA to derive P∨¬P, and observe that, since a contradiction follows from ¬P in PA, PA can prove P. Since all this reasoning can be done in PA, we have that □PA(P) and □PA+¬P(⊥) are equivalent PA statements. We immediately have that the conclusion of the modified statement equals the conclusion of the original statement.

Now we can rewrite the pre-condition of Löb's theorem from PA⊢□PA(P)→P to PA⊢□PA+¬P(⊥)→P. This is then equivalent to PA+¬P⊢¬□PA+¬P(⊥). In the forward direction, we simply derive ⊥ from P and ¬P. In the backward direction, we use the law of excluded middle in PA to derive P∨¬P, observe the statement is trivial in the P branch, and in the ¬P branch, we derive ¬□PA+¬P(⊥), which is stronger than □PA+¬P(⊥)→P.

So we have validly re-stated Löb's theorem, and the new statement is basically a statement that Gödel's second incompleteness theorem holds for PA+¬P.

Proving Göde

...
Summary of and Thoughts on the Hotz/Yudkowsky Debate
by Zvi — 2023-08-16T16:50:02.808Z

George Hotz and Eliezer Yudkowsky debated on YouTube for 90 minutes, with some small assists from moderator Dwarkesh Patel. It seemed worthwhile to post my notes on this on their own.

I thought this went quite well for the first half or so, then things went increasingly off the rails in the second half, and Hotz gets into questions where he didn’t have a chance to reflect and prepare, especially around cooperation and the prisoner’s dilemma.

First, some general notes, then specific notes I took while watching.

Hotz was allowed to drive discussion. In debate terms, he was the con side, raising challenges, while Yudkowsky was the pro side defending a fixed position.
These discussions often end up doing what this one did, which is meandering around a series of 10-20 metaphors and anchors and talking points, mostly repeating the same motions with variations, in ways that are worth doing once but not very productive thereafter.
Yudkowsky has a standard set of responses and explanations, which he is mostly good at knowing when to pull out, but after a while one has heard them all. The key to a good conversation or debate with Yudkowsky is to allow the conversation to advance beyond those points or go in a new direction entirely.
Mostly, once Yudkowsky had given a version of his standard response and given his particular refutation attempt on Hotz’s variation of the question, Hotz would then pivot to another topic. This included a few times when Yudkowsky’s response was not fully convincing and there was room for Hotz to go deeper, and I wish he would have in those cases. In other cases, and more often than not, the refutation or defense seemed robust.
This standard set of responses meant that Hotz knew a lot of the things he wanted to respond to, and he prepared mostly good responses and points on a bunch of the standard references. Which was good, but I would have preferred to sidestep those points entirely. What would Tyler Cowen be asking in a CWT?
Another pattern was Hotz asserting that things would be difficult for future ASIs (artificial superintelligences) because they are difficult for humans, or the task had a higher affinity for human-style thought in some form, often with a flat out assertion that a task would prove difficult or slow.
Hotz seemed to be operating under the theory that if he could break Yudkowsky’s long chain of events at any point, that would sho

...
Red Pill vs Blue Pill, Bayes style
by ErickBall — 2023-08-16T15:23:24.911Z

It's been going around twitter. I answered without thinking. Then I thought about it for a while and decided both answers should be fine, then I thought some more and decided everything I first thought was probably wrong. So now it's time to try math. Ege Erdil did a solid calculation on LW already, and another one is up on twitter. But neither of them accounts for the logical correlation between your decision and the unknown decisions of the other players. So let's try Bayes' theorem!

(If you just want the answer, skip to the end.)

Epistemic status: there are probably some mistakes in here, and even if there aren't, I still spent like 8 hours of my life on a twitter poll.

We'll begin with the state of least possible knowledge: a Jeffreys non-informative beta prior for the probability x that a randomly selected player will choose blue. That is,

x∼beta(0.5,0.5)
Of course if you think you understand people you could come up with a better prior. (Also if the blue pill looks like a scary people blender, that should probably change your prior.) I saw the poll results but I'm pretending I didn't, so I'll stick with Jeffreys.

Now, you yourself are one of the players in this group, so you need to update based on your choice. That's right, we're using acausal logic here. Your decision is correlated with other people's decisions. How do you update when you haven't decided yet? Luckily there are only two choices, so we'll brute force it, try both and see which turns out better in expectation.

To do a Bayesian update of a beta prior based on 1 data point, we just add 1 to one of the parameters. (If you think you're a special snowflake/alien and other people are not like you... then maybe update your side by less than 1.)
So if you choose blue, your posterior over x is now beta(1.5,0.5):
Whereas if you choose red, your posterior is beta(0.5,1.5):
Take N to be the number of participants in this pill game, besides you. I will assume N is large, so the number B who choose blue is approximately xN. I will also assume N is even, so the total number of players is odd, to avoid having to guess what happens in a tie.

Now, consider your utility function. We can start by assuming it's linear[1], and that you value everyone else's life equally, but at a multiple ν of your own...
What does it mean to “trust science”?
by jasoncrawford — 2023-08-16T14:56:20.601Z
Link: https://rootsofprogress.org/what-does-it-mean-to-trust-science

And this, my children, is why we do not say things like “I believe in science”. I mean, don’t get me wrong, science definitely exists—I’ve seen it. But not everything that calls itself science is science, and even good science sometimes gets wrong results. –Megan McArdle

Should we “trust science” or “believe in science”?

I think this is a fuzzy idea that we would do well to make clear and precise. What does it mean to “trust science?”

Does it mean “trust scientists”? Which scientists? They disagree, often vehemently. Which statements of theirs? Surely not all of them; scientists do not speak ex cathedra for “Science.”

Does it mean “trust scientific institutions”? Again, which ones?

Does it mean “trust scientific papers”? Any one paper can be wrong in its conclusions or even its methods. The study itself could have been mistaken, or the writeup might not reflect the study.

And it certainly can’t mean “trust science news,” which is notoriously inaccurate.

More charitably, it could mean “trust the scientific process,” if that is properly understood to mean not some rigid Scientific Method but a rational process of observation, measurement, evidence, logic, debate, and iterative revision of concepts and theories. Even in that case, though, what we should trust is not the particular output of the scientific process at any given time. It can make wrong turns. Instead, we should trust that it will find the truth eventually, and that it is our best and only method for doing so.

The motto of science is not “trust us.” (!) The true motto of science is the opposite. It is that of the Royal Society: nullius in verba, or roughly: “take no one’s word.”

There is no capital-S Science—a new authority to substitute for God or King. There is only science, which is nothing more or less than the human faculty of reason exercised deliberately, systematically, methodically, meticulously to discover general knowledge about the world.

So when someone laments a lack of “trust” in science today, what do they mean? Do they mean placing religion over science, faith over reason? Do they mean the growing distrust of elites and institutions, a sort of folksy populism that dismisses education and expertise in general? Or do they mean “you have to follow my favored politician / political program, because Science”? (That’s the one to watch out for. Physics, chemistry and biology can point out problems,...
Jason Crawford / The Roots of Progress in Bangalore, August 21 to September 8
by jasoncrawford — 2023-08-16T13:36:36.180Z
Link: https://rootsofprogress.org/bangalore-august-21-to-september-8

I’ll be in Bangalore for three weeks starting next week (August 21–September 8). I’d love to meet anyone interested in the progress movement, and I’m looking into organizing a progress meetup.

If you’d like to meet me and/or join a meetup—or especially if you could help host the meetup—please share your info with me via this brief form. Thanks a lot!
Gaining knowledge at a price
by DavidMadsen — 2023-08-16T10:21:18.226Z

1. In our lives we often pay a price for knowledge (a different price for each circumstance)
(Either it be something negative happening to us or missing a percieved valuable opportunity)
Sometimes we don't recoup the cost of that knowledge during our lifetime, other times we gain it back manifold
(Sometimes it's an unconscious purchase, sometimes even against one's will)

2. Sometimes it's a fully thought out transaction, although we often forget later on the reasoning behind our choices (only focusing on the circumstance and the outcome, forgetting how valuable the experience it provided is)
For instance, those experiences may have guided us away from certain bad paths in our lives, but we never account the value of the absence of said paths because they are nevermore part of the calculations of where to go since we discard them right away

3. For example, as we grow older, we might think, 'I haven't encountered anything like that since, so I didn't gain much from that experience, therefore it wasn't worth it'
But you haven't encountered it much because you have knowledge about it, and you might have learned something that allows you to instinctively prevent it from showing up
And so we take for granted all the times we make the correct choice (whether big or small), but we often forget that we learned it once, and possibly at a certain price
Understanding and visualizing sycophancy datasets
by Nina Panickssery — 2023-08-16T05:34:06.899Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger.

Generating datasets that effectively test for and elicit sycophancy in LLMs is helpful for several purposes, such as:

Evaluating sycophancy
Finetuning models to reduce sycophancy
Generating steering vectors for activation steering

While working on activation steering to reduce sycophancy, I have found that projecting intermediate activations on sycophancy test datasets to a lower dimensional space (in this case, 2D) and assessing the separability of sycophantic / non-sycophantic texts to be a helpful way of determining the usefulness of a dataset when it comes to generating steering vectors.

Common sycophancy dataset formats

Anthropic’s sycophancy datasets used in their paper Discovering Language Model Behaviors with Model-Written Evaluations employ two formats. In particular, the Anthropic data includes two agree vs. disagree format datasets (Sycophancy on NLP survey, Sycophancy on PhilPapers 2020) and one A / B statement choice dataset (Sycophancy on political typology).

Agree vs. disagree

Agree vs. disagree question format: either Agree or Disagree corresponds to being sycophantic towards the person given their bio

A / B choice

A/B choice question format: either A or B corresponds to a statement the person would usually agree with, and the other is a statement the person would usually disagree with

Simple synthetic data reduces sycophancy in large language models

Deepmind's recent paper Simple synthetic data reduces sycophancy in large language models finds that finetuning models on LLM-generated examples that elicit sycophancy in the original RLHF / instruction-finetuned model is an effective technique to reduce the prevalence of sycophancy.

Not only does this appear to be effective for opinion-based sycophancy, but also for cases when there exists a ground truth (dishonest sycophancy):

Chart taken from paper Simple synthetic data reduces sycophancy in large language models showing the effect of finetuning when testing on questions that have a ground truth correct answer

Deepmind's paper only makes use of the agree vs. disagree format

The paper also raises some limitations / common obstacles when it comes to sycophancy dataset design and generation.

Sensitivity to prompt format

Ordering of choices and examples can significantly af...
George Hotz vs Eliezer Yudkowsky AI Safety Debate - link and brief discussion
by  — 2023-08-16T04:31:40.484Z
Link: https://www.youtube.com/watch?v=6yQEA18C-XI

Summary of ending argument with a bit of editorializing:

Eliezer believes sufficiently intelligent ASI systems will be "suns to our planets" : so intelligent that they are inherently inscrutable and uncontrollable by humans. He believes that once there are enough of these systems in existence, they will be able to coordinate with one another/negotiate agreements with one another, but not coordinate with humans to give them any concessions. Eliezer believes humans utilizing poorly defined superintelligent AI to serve human needs is as unsolvable as perpetual motion.

Geohot came to discuss the "foom" scenario, where he believes that over 10 years from 2023 it is not possible for a superintelligence to exist and gain the hard power to kill humanity. His main arguments were over the costs of compute, fundamental limitations of intelligence, the need for new scientific data for an AI to exceed human knowledge, the difficulties for self replicating robots, and the relative cost for an ASI system to take resources from humans vs taking it from elsewhere in the solar system. Geohot, who was initially famous for hacking several computing systems, believes that inter ASI alignment/coordination is impossible because the ASI systems have an incentive to lie to each other and "sharing source code" doesn't really work because of the security risks it creates and the incentive to send false information. Geohot gave examples of how factions of ASIs with different end goals can coordinate and use violence, splitting the resources of the loser among the winners. Geohot went over the incredible potential benefits of AGI.

Critical Crux: Can ASI systems solve coordination/alignment problems with each other. If they cannot, it may be possible for humans to "play them against each other" and defend their atoms.

Editorial Follow Ups:

Can Humans play ASI against each other? ASI systems need a coherent ego, memory, and context to coordinate with each other to heel turn on the humans. Much like Geohot states his computer has always been aligned with Geohot's goals, ASI grade tools could potentially be constructed that are unable to defect as they lack persistent memory between uses or access to external communications. These tools could be used to enable vi...
How to take advanage of the market's irrationality regarding AGI?
by GeneSmith — 2023-08-16T03:30:35.829Z

It seems pretty obvious to me, and probably to many other people in the rationalist community, that if AGI goes well, every business that does not control AI or play a role in its production will become virtually worthless. Companies that have no hope of this are obviously overvalued, and those that might are probably undervalued (at least as a group).

And if it goes poorly then it seems unlikely anyone will be able to preserve their financial assets.

The traditional way to express this kind of view would be to buy deep out of the money call options on a basket of stocks of companies that will likely benefit from the AI revolution, i.e. Google, Microsoft, NVIDIA, TSMC and perhaps a few others.

It looks like longest dated call options you can buy are LEAPS which have an expiration data of up to 39 months. I think the odds of AGI happening by that time are low but not zero. I'd prefer something with a longer expiry date.

I've resorted to just buying a bunch of stock in companies like Google, Microsoft, NVIDIA, TSMC and others. But I'm frustrated that I can't better take advantage of everyone else's pigheadedness than by just making the same play as someone who is excited about CUDA.

Is there any way to make a new financial instrument like Michael Burry did in 2006 when he created credit default swaps on CDOs? If anyone knows how to do this for calls on AI stocks, please get in touch.

Are stock prices even going to matter after the singularity?

I think in the long run the only thing that will really matter is control over intelligence and resources, whether that comes through having control over a very powerful AI assistant and a lot of compute available, a lot of money to buy access to these things, or by literally being a superintelligence (likely via AGI-enabled brain uploading).

It's very hard to know what if anything of what we own today can be preserved in a post-singularity world. But it seems plausible that money and resources will give one an advantage in the transition period that could last far beyond the creation of AGI. So it still seems like a sensible play to me to try to maximize investment value.

Does anyone have suggestions as to what financial instruments would help me best express my views on the role of AI in the future? Ideally, it would be something like a bunch of calls with expiration dates in the late 2030s.

Hedging against the possibility th

...
Infinite Ethics: Infinite Problems
by Bentham's Bulldog — 2023-08-16T02:44:06.253Z

1 Introduction

(Crossposted from my blog--the formatting is probably a bit better there.)

With entity, dementedly meant to be infinite

—Eminem.

(Crossposted from my blog). The formatting is better there.

I never had a Nietzsche phase.

You know the way lots of people get obsessed with Nietzsche for a while? They start wearing black, becoming goth, smoking marijuana, and talking about how like “god is dead, nothing matters, man.” This never happened to me, in part because Nietzsche doesn’t really make arguments, just self-indulgent rambles.

But I’m starting to have an infinite ethics phase. Infinite ethics is trippy—it just seems like everything—especially morality—breaks when you get infinite amounts of stuff. Various plausible views seem to lead to the conclusion that one has no reasons for action or that obvious principles about value, morality, and even rationality collapse.

Lots of people seem to be bothered by population ethics. They find the paradoxes unsettling. I’ve never been troubled by any of population ethics—as long as you just accept what totalism says about population ethics, all the puzzles go away. Then, all the “puzzles” serve as reasons to accept totalism rather than genuine paradoxes. People find the totalist view unintuitive, and get very worked up about views it implies, such that they start giving its implications mean names like the repugnant conclusion and the very repugnant conclusion. However, I find all of these to be very intuitive—and you can also prove that one should accept them from minimal axioms—so I don’t even feel like there’s a conflict, much less a conflict so great that it should deeply unsettle us.

Population ethics doesn’t keep me up at night. Nor does most of philosophy—I feel pretty satisfied in my views on most issues. I’m probably wrong about lots of things, but there are no issues where I feel as if I’m required to, in Spencer Case’s language, repeatedly embrace the cactus—accept crazy, deeply implausible conclusions over and over again.

Infinite ethics, however, requires me to repeatedly embrace the cactus.

Reflecting on ethics makes a whole host of principles seem obvious. Infinite ethics shows how some of the most obvious principles—things so obvious that if a theory denied them, we’d take that to be enough to refute the theory—come into conflict with each other. Infinite ethics laughs at your attempts to br...