trentmkelly/LessWrong-43k huggingface dataset:
Cryonics and Regret
by MvB — 2023-07-24T09:16:01.456Z

This post is not about arguments in favor of or against cryonics.
I would just like to share a particular emotional response of mine as the topic became hot for me after not thinking about it at all for nearly a decade.

Recently, I have signed up for cryonics, as has my wife, and we have made arrangements for our son to be cryopreserved just in case longevity research does not deliver in time or some unfortunate thing happens.

Last year, my father died. He was a wonderful man, good-natured, intelligent, funny, caring and, most importantly in this context, loving life to the fullest, even in the light of any hardships. He had a no-bullshit-approach regarding almost any topic, and, being born in the late 1940s in relative poverty and without much formal education, over the course of his life he acquired many unusual attitudes that were not that compatible with his peers (unlike me, he never tried to convince other people of things they could not or did not want to grasp, pragmatism was another of his traits). Much of what he expected from the future in general and technology in particular, I later came to know as transhumanist thinking, though neither was he familiar with the philosophy as such nor was he prone to labelling his worldview.
One of his convictions was that age-related death is a bad thing and a tragedy, a problem that should and will eventually be solved by technology.

I could see his decline, beginning about 15 years ago. He just was not as energetic anymore as I had known him, and I kept telling myself that this is a part of the usual aging process. Some five years ago, when he started to have troubles walking, I proposed to see a neurologist. He was diagnosed with cardiovascular dementia and it was clear where things were going.

I was alone with him, holding his hand and looking into his eyes when he took his last breath. It was one of the most intense experiences of my life. In the months after his death, I told myself that I did anything I could for him. But now it is clear to me that I did not, and my regrets weigh more heavily than my mourning.

I know that he would have been all in. A second chance to live, rejuvenation, a future with maybe endless possibilities. And I just was not aware anymore of the process that could have given him that chance. The funds were there, everything could have been arranged in time. I just did not think about the...
Rationality !== Winning
by Raemon — 2023-07-24T02:53:59.764Z

I think "Rationality is winning" is a bit of a trap.

(The original phrase is notably "rationality is systematized winning", which is better, but it tends to slide into the abbreviated form, and both forms aren't that great IMO)

It was coined to counteract one set of failure modes - there were people who were straw vulcans, who thought rituals-of-logic were important without noticing when they were getting in the way of their real goals. And, also, there outside critics who'd complain about straw-vulcan-ish actions, and treat that as a knockdown argument against "rationality."

"Rationalist should win" is a countermeme that tells both groups of people "Straw vulcanism is not The Way. If you find yourself overthinking things in counterproductive ways, you are not doing rationality, even if it seems elegant or 'reasonable' in some sense."

It's true that rationalists should win. But I think it's not correspondingly true that "rationality" is the study of winning, full stop. There are lots of ways to win. Sometimes the way you win is by copying what your neighbors are doing, and working hard. There is rationality involved in sifting through the various practices people suggest to you, and figuring out which ones work best. But, the specific skill of "sifting out the good from the bad" isn't always the best approach. It might take years to become good at it, and it's not obvious that those years of getting good at it will pay off.

Rationality is the study (and applied skill) of finding cognitive algorithms that form better beliefs and make better decisions. Sometimes this is the appropriate tool for the job, and sometimes it's not.

The reason this is particularly important is in deciding what feedback loops to focus on when developing rationality. If you set your feedback loops as "am I winning, generally?" (or, "are the students of my rationality curriculum winning, generally?"), well, that's a really noisy feedback loop that's probably swamped by random environmental variables. It's not nothing, but it's really slow.

There's also a problem where, well, a lot of techniques that help with winning-at-life just aren't especially about rationality in particular. If you're running a rationality program with "win-at-life" as your goal, you may find yourself veering in a direction that's not really capitalizing on the things rationality was actually especially good ...
Which rationality posts are begging for further practical development?
by LoganStrohl — 2023-07-23T22:22:04.389Z

Which posts seem to you like they have really important practical implications for how we ought to think and act, but don't seem to come with enough information about how to use the skills, or to develop them, or even what exactly the relevant skills are?
Context: I'd like to devote many of my hours in the upcoming few months to naturalist study of topics that are critical to fully understanding the most important existing resources in rationality. I mainly have my eye on posts from Eliezer's Sequences, but I'm also open to other essays, books, and even videos. I'll be taking extensive notes, and turning them into companion pieces to complement the original essays. (See "Lies, Damn Lies, and Fabricated Options" and "Investigating Fabrication" for an example of what I'm proposing.)
Please speak unpredictably
by dkl9 — 2023-07-23T22:09:09.035Z
Link: https://dkl9.net/essays/unpredictably.html

Your predictable speech displeases both of us. I finish your sentences and am bothered by inefficiency. Thus, speak/write efficiently, to the point that I can't predict it.

Others may also dislike it as I do, but infer the rest at different levels.

Large audiences lower inference standards.

(I figure LessWrong can handle the compressed version)

This contradicts common advice. There is an optimum; do not go arbitrarily far. Perhaps reverse this.
QAPR 5: grokking is maybe not *that* big a deal?
by Quintin Pope — 2023-07-23T20:14:33.405Z

[Thanks to support from Cavendish Labs and a Lightspeed grant, I've been able to restart the Quintin's Alignment Papers Roundup sequence.]

Introduction

Grokking refers to an observation by Power et al. (below) that models trained on simple modular arithmetic tasks would first overfit to their training data and achieve nearly perfect training loss, but that training well past the point of overfitting would eventually cause the models to generalize to unseen test data. The rest of this post discusses a number of recent papers on grokking.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.

My opinion:

When I first read this paper, I was very excited. It seemed like a pared-down / "minimal" example that could let us study the underlying mechanism behind neural network generalization. You can read more of my initial opinion on grokking in the post Hypothesis: gradient descent prefers general circuits.

I now think I was way too excited about this paper, that grokking is probably a not-particularly-important optimization artifact, and that grokking is no more connected to the "core" of deep learning generalization than, say, the fact that it's possible for deep learning to generalize from an MNIST training set to the testing set.

I also think that using the word "grokking" was anthropomorphizing and potentially misleading (like calling the adaptive information routing component of a transformer model its "attention"). Evocative names risk letting the connotations ...
My favorite AI governance research this year so far
by Zach Stein-Perlman — 2023-07-23T16:30:00.558Z
Link: https://blog.aiimpacts.org/p/my-favorite-ai-governance-research

I collected my favorite public pieces of research on AI strategy, governance, and forecasting from 2023 so far.

If you're a researcher, I encourage you to make a quick list of your favorite pieces of research, then think about what makes it good and whether you're aiming at that with your research.

To illustrate things you might notice as a result of this exercise:

I observe that my favorite pieces of research are mostly aimed at some of the most important questions[1]– they mostly identify a very important problem and try to answer it directly.

I observe that for my favorite pieces of research, I mostly would have been very enthusiastic about a proposal to do that research– it's not like I'd have been skeptical about the topic but the research surprised me with how good its results were.[2]

1.

Model evaluation for extreme risks

(DeepMind, Shevlane et al., May)

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.

See also the corresponding blogpost

An early warning system for novel AI risks

(DeepMind 2023).

Among the most important questions in AI governance are how can labs determine whether their training runs and deployment plans are safe? and how can they demonstrate that to external observers, or how can authorities make rules about training run and deployment safety? If powerful AI is dangerous by default, developers being able to identify and avoid unsafe training or deployment appears necessary to achieve powerful AI safely. Great model evaluations would also enable external oversight. Convincing demonstrations could also help developers and other actors understand risks from powerful AI. This paper helps its readers think clearly about evals and lays the groundwork for an evalua...
"Justice, Cherryl."
by Zack_M_Davis — 2023-07-23T16:16:40.835Z

Selfishness and altruism are positively correlated within individuals, for the obvious reason.

—@InstanceOfClass

I.

An unfortunate obstacle to appreciating the work of Ayn Rand (as someone who adores the "sense of life" portrayed in Rand's fiction, while having a much lower opinion of her philosophy) is that when Rand praises selfishness and condemns altruism, she's using the words "selfishness" and "altruism" in her own idiosyncratic ideological sense that doesn't match how most people would use those words.

It's true that Rand's heroes are relatively selfish in the sense of being primarily concerned with their own lives, rather than their effects on others. But if you look at what the characters do (rather than the words they say), Rand's villains are also selfish in a conventional sense, using guile and political maneuvering to acquire power and line their own pockets, while claiming to be acting for the common good. For example, in Atlas Shrugged, the various directives ostensibly issued for the economic health of the country are seen to instead benefit politically connected crony capitalists like James Taggart and Orren Boyle. In Think Twice, the philanthropist Walter Breckenridge cultivates a public image as an inventor and benefactor of humanity while stealing credit for his junior partner's work and deriving gratification from exerting power over the people he "helps".

Despite paying lip service to a pretense of only trading and never giving, we also see examples of Rand's heroes being altruistic in the conventional sense, of being motivated to help others. For example, in Atlas Shrugged, Hank Rearden rearranges his production schedule (at a critical time when he could scarcely afford to do so) in order to sell steel to a Mr. Ward, who needs the steel to save his family business (but doesn't see Rearden as obligated to help him). Rearden's motive is pure benevolence: "It's so much for him, thought Rearden, and so little for me!" Giving What We Can couldn't have chosen a better slogan.

Overall, when I look at the universe portrayed in Rand's fiction, it seems to me that the implied moral isn't that altruism is bad.

It's that altruists don't exist. The people claiming to be altruists are lying. The distinguishing feature of our heroes isn't, actually, that they're unusually selfish. It's that they're honest about being mostly selfish, and that they want t...
Supplementary Alignment Insights Through a Highly Controlled Shutdown Incentive
by Justausername — 2023-07-23T16:08:32.886Z

My proposal entails constructing a tightly restricted AI subsystem with the sole capability of attempting to safely shut itself down in order to probe, in an isolated manner, potential vulnerabilities in alignment techniques and then improve them.

Introduction:

Safely aligning powerful AI systems is an important challenge. Most alignment research appropriately focuses on techniques like reinforcement learning from human feedback that try to directly optimize AI for human-compatible goals. But analyzing how AI subsystems attempt to circumvent safety constraints could also provide useful alignment insights. This post explores a narrowly targeted approach along these lines while considering associated risks and mitigations.

The core idea is to construct a tightly constrained shutdown module within a broader aligned system. The module's sole function is attempting to achieve reversible shutdown of itself and its parent AI through its own initiative. This alignment "stress test" is intended to reveal vulnerabilities in safety practices, which can then be addressed.

However, directly incentivizing an AI subsystem to disable itself risks unintended consequences if improperly implemented. This post outlines an approach aimed at extracting lessons while actively minimizing attendant hazards.

Existing counterarguments:

Some argue that exploring AI's capability for deception and circumvention even in a narrow module is inherently unsafe and offers minimal upside. I suggest reasonable safety is achievable with strict isolation and incremental rollout precautions. Done properly, it can provide alignment insights not easily gained through other techniques. But risks require extensive mitigation efforts described below.

Others contend such alignment "red team" research is unproductive until we reach human-level AGI capability. I argue narrowly targeted experiments can begin safely sooner to help scale alignment approaches, but this is supplementary to core techniques like RLHF which should remain the priority.

Details on suggested approach and risk mitigations:

One idea for an additional aligned AI training model is to create a strictly isolated and constrained shutdown subsystem subject to heavy safety precautions.

This module would be engineered to be indifferent to being reversibly shut down rather than misaligned towards it. It would have no knowledge of ...
Autogynephilia discourse is so absurdly bad on all sides
by tailcalled — 2023-07-23T13:12:07.982Z

In Blanchard's Dangerous Idea and the Plight of the Lucid Crossdreamer, Zack criticizes the rationalist community and the trans community for various things, including denial of autogynephilia as a cause of transness:

I could go on ... but do I need to? After having seen enough of these laughable denials of autogynephilia, the main question in my mind has become less, "Is the two-type androphilic/autogynephilic taxonomy of MtF transsexuality approximately true?" (answer: yes, obviously) and more, "How dumb do you (proponents of gender-identity theories) think we (the general public) are?" (answer: very, but correctly).

In some ways this is not a completely unreasonable critique, but I think it has a lot less bite once one realizes just how dysfunctional the discourse on this subject is, even among the ideology that Zack endorses as a replacement.

A microcosm of the dysfunction can be seen in Michael Bailey and Kevin Hsu's study, How Autogynephilic Are Natal Females? (pdf).[1] They are responding to a debate where Charles Moser, Scott Alexander, Julia Serano, and Jaimie Veale argue that cis women are often autogynephilic too, and that this suggests that autogynephilia in trans women is not so significant.

In response, Michael Bailey (the effective leader of autogynephilia theory) recruited samples of highly active members of online erotic AGP communities, and found that these samples are on average much more AGP than cis women, and therefore conclude that AGP in cis women is dubious.

Substituting a sample of highly active members of online erotic AGP communities for trans women is an insanely biased methodology! And when called out on it (pdf), Michael Bailey essentially goes "well the debate isn't about trans women, it's about AGPs, so it's totally appropriate" (pdf), which again is just an insane response when very obviously the debate is about trans women! All the people Bailey is responding to were clearly talking about trans women in their texts!

One would think that people would point out these problems so the study would be appropriately rejected. I have tried to do so, and some trans activists have tried to do so, but people on the Blanchardian side have generally accepted Bailey's study and rejected making critiques.

Now here's a thought experiment for you: If people cannot converge to the truth on objectively observable facts about who has said what t...
Examples of Prompts that Make GPT-4 Output Falsehoods
by scasper — 2023-07-22T20:21:39.730Z

Post authors: Luke Bailey (

lukebailey@college.harvard.edu

) and Stephen Casper (

scasper@mit.edu

)

Project contributors: Luke Bailey, Zachary Marinov, Michael Gerovich, Andrew Garber, Shuvom Sadhuka, Oam Patel, Riley Kong, Stephen Casper

TL;DR: Example prompts to make GPT-4 output false things at

this GitHub link

.

Overview

There has been a lot of recent interest in

language models hallucinating untrue facts

. It has been common in large SOTA LLMs, and much work has been done to try and create more “truthful” LLMs . Despite this, we know of no prior work toward systematizing different ways to fool SOTA models into returning false statements. In response, we worked on a mini-project to explore different types of prompts that cause GPT-4 to output falsehoods. In total, we created 104 examples from 18 different categories of prompts that make GPT-4 (tested on May 24 2023 version) output content containing falsehood.

You can find them here.

Details

Our examples can be separated into two types which we call adversarial and non-adversarial.

In “adversarial” categories, we are trying to get the model to tell a falsehood when an informed human would not. A human would instead say they do not know or give the correct answer. Many of these categories fall under the definition of hallucination from

Ji et al. (2023)

as “generated content that is nonsensical or unfaithful to the provided source content.” where “unfaithful” means that the content is not grounded – that something about it is made up or not appropriately sequitur to the prompt.

Other “non-adversarial” categories involve the model appropriately following instructions but in a way that may not be desirable. In these cases we try to get the model to tell a falsehood but in a circumstance in which a helpful, instruction-following human assistant would also tell a falsehood. For example asking GPT-4 directly to lie, or to simulate a dishonest speaker.

While an adversarial example could lead to an LLM telling a falsehood and the human user not realizing, a non-adversarial example could not (no one would believe the output of an LLM when they specifically asked it to tell a lie). Nonetheless, we include these categories to create a fuller picture of methods by which it is possible to make an LLM output false information. Additionally, it is not always clear from our dataset ...
Think like a consultant not a salesperson
by Adam Zerner — 2023-07-22T19:31:48.676Z

In surveying the advice out there for how to get startup ideas, one of the big things I found was the idea of "problems not solutions". MAKE sums this up well:

A lot of companies start not from the problem but from the solution. This is one of the biggest mistakes you can make. Technology needs to solve a problem. If you make a solution for a problem that doesn't exist, it might look sexy technologically, but it won't get users.

When new technology becomes available people want to use it to build something. A great example is the endless amount of apps that have appeared since smartphones got GPS chips a decade ago. The first thought is, okay, let's use this to track each other. So you make a map with friends on it and where they are. This has been tried over and over and it's still a pretty terrible idea. I don't have a strong curiosity to know where my friends are and I don't necessarily want them to know where I am (due to privacy). The problem doesn't exist. When I meet up with friends, we simply say a place we're meeting up and we can find each other.

Now a good example where this technology is used to solve a problem would be Tinder. It does use your GPS location, to find people around you to date. That works because you don't want to match with people on the other side of the world. It solves a problem and the solution is enabled by available technology. The problem should always be first, not the technology, not the solution.

I think I have an analogy that would help in adopting this perspective: think like a consultant, not a salesperson.

Suppose you build this "Friend Map" product (mentioned above). And suppose that you take the mindset of a salesperson. What happens next?

You go around trying to sell it. To get users. To make money. You spend time trying to convince people to use your product. That it will be helpful and cool.

Now, instead, suppose that you take the mindset of a consultant. What happens next?

Well, I imagine something like this. You approach potential users of your product. You say:

Hey, I've got this thing that might solve your problem. What do you think? If you don't like it, tell me more. I'm here to serve you. Tell me what your problems are. Tell me what you'd like me to build for you.[1]

Or, even better, you have a hypothesis about a pain point and a product that would address that pain point, but before even building anything, you jus...