trentmkelly/LessWrong-43k huggingface dataset:
How have you become more hard-working?
by Chi Nguyen — 2023-09-25T12:37:39.860Z

I'd be curious to hear stories of people who have successfully become more hard-working, especially if they started out as not particularly hard-working. Types of things I can imagine playing a role or know have played a role for some people:

Switching roles to something that is conducive to hard work, e.g. a fast-paced environment with lots of concrete tasks and fires to put out.
Medication, e.g. ADHD medication
Internal work, e.g. specific types of therapy, meditation, self-help reading, or other types of reflection.
Productivity hacks, e.g. more accountability, putting specific systems in place
Motivational events, arguments, or life periods, e.g. working a normal corporate jobs where long hours are expected
Switching work environment to something that is conducive to hard work, e.g. always working in an office with others who hold you accountable

This curiosity was triggered by realising that I know of very few people that have become substantially harder-working over their late adolescence/adult life. I also noticed that the few people that I know successfully and seemingly permanently increased their mental health/work satisfaction always were hard-working even when they were unhappy (unless they were in the middle of burn-out or similar).

People becoming more hard-working seems really useful but I haven't seen much in terms of evidence that it's feasible or effective methods. If there are books or studies on this topic, those would also be welcome. Thank you!
Automating Intelligence: A Cursory Glance at How AutoML Brings Precision to AI Development
by RoscoHunter — 2023-09-25T09:39:31.338Z

Motivation: At the outset of 2023, OpenAI unveiled GPT-4 [1], claiming it to be "at or beyond human-level for many tasks... showing sparks of artificial general intelligence". This mirrors the breakthrough that marked the start of the 20th century when the Wright Brothers achieved the first sustained and controlled heavier-than-air powered flight. In this article we will draw further parallels between artificial flight and intelligence to frame our progress on AI and offer insights into what’s to come. For example, early models in both fields were inspired by nature and limited to the most rudimentary of abilities. Aviation's pioneers drew inspiration from bird wings for early gliders, and similarly, AI developers were initially inspired by biological neurons.

After years of incremental advancements, the Wright Brothers and OpenAI demonstrated humanity's capacity to compete with nature for the very first time — in the realms of flight and intelligence, respectively. Following the Wright brothers’ flight, aviation witnessed a surge of investment that turned aircraft production into a science. After AI’s “Chat-GPT” moment a similar precision is now required to fan the sparks of AGI. Enter Automated Machine Learning (AutoML), which aims to systematize and streamline the design of AI models. In this article, we will delve into two instances of AutoML, highlighting its impact on architecture design and learning algorithms. Hopefully, this article will provide a cursory glance at how AutoML is revolutionizing AI development, elevating it from the realm of hobbyist aircraft design to the precision of sonic jet production.

Architecture Design: Traditionally, AI models were hand-designed based on the judgments of seasoned AI researchers. A branch of AutoML, Neural Architecture Search (NAS), challenges this setup with a more rigorous framework, using methods like reinforcement learning [3] or gradient descent [4] to locate high-performance networks from a space of possible designs. Instead of relying on intuition or tedious human-made experiments with different layer sizes and configurations [5], NAS automates the entire procedure by efficiently searching through the innumerate ways that network components can be combined. However, it still typically requires training the networks that it samples during search, thereby slowing the pace of de...
Interpreting OpenAI's Whisper
by EllenaR — 2023-09-24T17:53:44.955Z

(Work done as part of SERI MATS Summer 2023 cohort under the supervision of @Lee Sharkey . A blog post containing audio features that you can listen to can be found here.)

TL;DR - Mechanistic Interpretability has mainly focused on language and image models, but there's a growing need for interpretability in multimodal models that can handle text, images, audio, and video. Thus far, there have been minimal efforts directed toward interpreting audio models, let alone multimodal ones. To the best of my knowledge, this work presents the first attempt to do interpretability on a multimodal audio-text model. I show that acoustic features inside OpenAI's Whisper model are human interpretable and formulate a way of listening to them. I then go on to present some macroscopic properties of the model, specifically showing that encoder attention is highly localized and the decoder alone acts as a weak LM.

Why we should care about interpreting multimodal models

Up to this point, the main focus in mechanistic interpretability has centred around language and image models. GPT-4, which currently inputs both text and images, is paving the way for the development of fully multimodal models capable of handling images, text, audio, and video. A robust mechanistic interpretability toolbox should allow us to understand all parts of a model. However, when it comes to audio models, let alone multimodal ones, there is a notable lack of mechanistic interpretability research. This raises concerns, because it suggests that there might parts of multimodal models that we cannot understand. Specifically, an inability to interpret the input representations that are fed into the more cognitive parts of these models (which theoretically could perform dangerous computations) presents a problem. If we cannot understand the inputs, it is unlikely that we can understand the potentially dangerous bits.

This post is structured into 3 main claims that I make about the model:

The encoder learns human interpretable features
Encoder attention is highly localized
The decoder alone acts as a weak LM

For context: Whisper is a speech-to-text model. It has an encoder-decoder transformer architecture as shown below. We used Whisper tiny which is only 39M parameters but remarkably good at transcription! The input to the encoder is a 30s chunk of audio (shorter chunks can be padded) and the output from the decoder is...
Contradiction Appeal Bias
by onur — 2023-09-24T17:03:58.724Z

Contradiction Appeal Bias, or Counterintuitivity Bias, is the tendency to believe statements that contradict common knowledge simply because they're unexpected. Think conspiracy theories, health myths and so on.

Causes

Several factors drive this bias, including but not limited to:

Self-Doubt: Individuals unsure of their own understanding may be more easily swayed by contradictory information.

Emotion: Alarming claims, especially those that trigger fear, can make people believe without much evidence.

Information Overload: When bombarded with information, counterintuitive claims might be accepted without scrutiny.

Examples

Flat Earth Theory: Despite the scientific consensus on a spherical Earth, some are drawn to the flat Earth idea because it starkly contradicts mainstream thought. Though I do not purport that this idea alone explains the existence of conspiracy theories.

Health Myths: Claims like "drinking cold water after meals causes cancer" can spread rapidly, not due to scientific evidence, but because they challenge common practices and beliefs.

This is a rough first sketch. Looking for quick feedback on the idea, will update in the future.
RAIN: Your Language Models Can Align Themselves without Finetuning - Microsoft Research 2023 - Reduces the adversarial prompt attack success rate from 94% to 19%!
by Singularian2501 — 2023-09-24T16:48:18.360Z

Paper: https://arxiv.org/abs/2309.07124

Abstract:

Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, the so-called finetuning step. In contrast, aligning frozen LLMs without any extra data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide backward rewind and forward generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates; during the self-evaluation phase, the model receives guidance on which human preference to align with through a fixed-template prompt, eliminating the need to modify the initial prompt. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B over vanilla inference from 82% to 97%, while maintaining the helpfulness rate. Under the leading adversarial attack llm-attacks on Vicuna 33B, RAIN establishes a new defense baseline by reducing the attack success rate from 94% to 19%.
Honor System for Vaccination?
by jefftk — 2023-09-24T11:50:05.809Z

In June the social dance I help organize decided to

drop
its requirement that people be vaccinated and boosted:

Checking vaccination cards at the door requires an additional
volunteer, is a hassle for attendees, and at this point the
communities we draw from are overwhelmingly vaccinated. While we
still recommend staying up to date on vaccinations (including your flu
shot and covid booster in the fall!), we're dropping this requirement
starting with our July 2nd dance.

One question we got, both in person and electronically, was why not
keep requiring vaccination on the honor system? Speaking for myself
and not BIDA, I think this is rarely a good policy for events.

(I'm setting aside the question of whether vaccination is helpful in
reducing risk to others, and writing all of this under the assumption
that it is. I do think this is unclear, especially if a policy does
not require a recent booster.)

The issue is, an honor system is a bad fit when getting caught is very
unlikely and the harms of violating the rule are diffuse and unclear.
If an unvaccinated dancer ignores the policy and does happen to bring
an infection to the dance that vaccination would have prevented, it's
very unlikely that anyone will ever know. There's no feedback
mechanism promoting honesty here. Instead, you're filtering
unvaccinated people for ones who are ok violating the policy, and
nudging more unvaccinated people toward being dishonest in this way.

Compare this to a dance in a small community that does payment on the
honor system: an unattended fiddle case by the door for you to drop
your money into. If someone doesn't contribute it's much more likely
that people will notice, and it's also much clearer that when you
defect you're hurting people. There's still some pressure toward
dishonesty, but there's enough pushing the other way that it still
works well.

The honor system can work well in some cases, but I don't think this
is one of them.

Comment via: facebook, mastodon
Far-Future Commitments as a Policy Consensus Strategy
by FCCC — 2023-09-24T06:34:55.505Z

In government, it’s not just a matter of having the best policy; it’s about getting enough votes. This creates a problem when the self-interests of individual voters don’t match the best interests of the country.

For instance, voting researchers widely consider the presidential voting system in America to be inferior to many alternatives. But if you want to change it, you require consent from Democrats and Republicans—i.e. the very people who benefit from the status quo.

Or consider the land-value tax. This tax is considered among economists to be uniquely efficient (i.e. it causes zero reduction in the good being taxed). When implemented correctly, it can address edge cases, such as new property developments, and can even prevent reductions in new land production, like the creation of artificial islands. However, this policy is against the interests of current land owners. So any politician who advocates this policy would not only fail but also likely end their political career.

What do these policies have in common? Well, both policies yield long-run benefits, and as we’ll see, they impose short-run costs. (If you disagree that these policies are actually beneficial in the long run, I’m sure you can think of policies that you like that have long-run benefits and short-run costs. The examples I’ve given are simply for clarity.)

What if rather than asking American politicians to vote against their own interests, we ask them to pass a policy today that will only be enacted after a 100-year period? Significant advances in medicine notwithstanding, by that time, most of today’s politicians will be dead. In other words, they no longer have to vote against their own interests.

The same strategy can be applied to land value taxes. If today, we passed a policy for a 100-year delayed land-value tax, the effect on house prices is approximately zero (see the widely-accepted net present value model of asset prices).

I believe this strategy offers a significant opportunity. Policy in the EA community is often seen as too hard because of how crowded it is. Yet almost no policy makers are thinking about the future in 100 years. It might be possible to pass a lot of “unpassable” policies. We just have to ensure the policies we propose are actually good (see my unfinished series on the topic), and have large barrier to reversal, so that the politicians of the future can’t renege ...
Five neglected work areas that could reduce AI risk
by  — 2023-09-24T02:03:29.829Z

Tldr: We identify five areas of work that should be further investigated:

Helping information aggregators scale during advanced AI development,
Improving internal AI deployment in policy organizations,
Researching the institutional design for evaluating alignment plans,
Planning for how to evaluate automated alignment researchers, and
Creating further education and information material.

Summary

We outline and discuss five areas of work that seem neglected. We only shallowly review and discuss these areas. Our proposals should be seen as a recommendation to further investigate these questions and figure out whether they are actually worth pursuing and how to pursue them.

For each of the five work areas, we discuss what they are, what one could do here, why they are important, and what our key uncertainties are.

Help Information Aggregators and Reviewers Scale during Takeoff

Effective information provision and compilation are vital for informed societal choices. As advanced AI systems evolve, current information channels and curators, such as journalists, social media platforms, think tanks, and governments, will likely struggle to keep up. Identifying ways to support them or introducing new organizations to monitor global events seems important.

Consult and Research Responsible AI deployment in Governments

Decision-making bodies such as the government are facing the increasingly important question of how to deploy advanced AI systems internally. This decision can make societal decision making much better but also much worse. Research and advising of policymakers can help avert various failure modes of internal AI deployment.

Research Institutional Design for Evaluating Alignment Plans

Many AI governance plans involve a public agency evaluating the alignment plan for an advanced AI project and then approving or rejecting the training run. Designing such an evaluation process seems hard, and default institutional designs seem likely insufficient, so somebody should figure out what this process should ideally look like.

Plan for How to Evaluate Automated Alignment Researchers

Most of the alignment work might be conducted by AI systems. OpenAI, for instance, has plans to develop automated alignment researchers. However, the success of this relies on being able to successfully evaluate alignment work at scale, which is currently lacking. We need to figure

...
Are the other Rationality: A-Z sequences coming out as books?
by caffeinated_dissonance — 2023-09-24T00:38:51.939Z

I found a website that purports that the remaining 4 books in the series are “coming soon.” But it doesn’t indicate a date, and I don’t know if they’re perpetually “coming soon” or if this is still a project being pursued. Anyone know about the status of these books?
The Dick Kick'em Paradox
by Augs SMSHacks — 2023-09-23T22:22:06.827Z

What is the point of this thought experiment?

To demonstrate that thought experiments involving mind reading agents (such as the AI in "Newcomb's Paradox" or Paul Ekman) can create scenarios in which any decision theory will be worse off than another. If we allow these agents to exist then no decision theory will be stricly better than anther.

The Paradox

Let's say we have two competing decision theories ADT and BDT, these can be any two decision theories you like, so long as they are different. Here, different is meant that there exists a scenario where ADT leads to a different decision than BDT.

Consider a mind-reading agent like Paul Ekman or a super intelligent AI that we will call Dick Kick'em. Like in the thought experiments mentioned above, Dick Kick'em can read the minds of other agents, know their beliefs, and future actions with near certainty.

Now imagine you are wandering the streets and you come across Dick Kick'em. He says "I am going to read you mind and if you believe in ADT I will leave you alone, but if you believe in BDT I will kick you in the dick". Let's consider the scenarios:

You believe in ADT: You get left alone, net zero utility.

You believe in BDT: You get kicked in the dick, negative utility.

Therefore, ADT is a superior decision theory compared to BDT.

Conclusion

Since ADT and BDT are arbitrary decision theories, we can use this scenario to show that CDT is better than UDT or even TDT. Either that, or the idea of mind reading agents is flawed. Thought experiments like "Newcomb's Paradox" essentially boil down to the same thing as the "Dick Kick'em Paradox" except instead of having the AI kick you directly there is a small game that makes it look like there's actual decision theory involved.
I designed an AI safety course (for a philosophy department)
by Eleni Angelou — 2023-09-23T22:03:00.036Z

Background

In the fall of 2023, I'm teaching a course called "Philosophy and The Challenge of the Future"[1] which is focused on AI risk and safety. I designed the syllabus keeping in mind that my students:

will have no prior exposure to what AI is or how it works
will not necessarily have a strong philosophy background (the course is offered by the Philosophy department, but is open to everyone)
will not necessarily be familiar with Effective Altruism at all

Goals

My approach combines three perspectives: 1) philosophy, 2) AI safety, and 3) Science, Technology, Society (STS); this combination reflects my training in these fields and attempts to create an alternative introduction to AI safety (that doesn't just copy the AISF curriculum). That said, I plan to recommend the AISF course towards the end of the semester; since my students are majoring in all sorts of different things, from CS to psychology, it'd be great if some of them considered AI safety research as their career path.

Course Overview

INTRO TO AI

Week 1 (8/28-9/1): The foundations of Artificial Intelligence (AI)

Required Readings:

Artificial Intelligence, A Modern Approach, pp. 1-27, Russell & Norvig.
Superintelligence, pp. 1-16, Bostrom.

Week 2 (9/5-8): AI, Machine Learning (ML), and Deep Learning (DL)

Required Readings:

You Look Like a Thing and I Love You, Chapters 1, 2, and 3, Shane.
But what is a neural network? (video)
ML Glossary (optional but helpful for terminological references)

Week 3 (9/11-16): What can current AI models do?

Required Readings:

Artificial Intelligence, A Modern Approach, pp. 27-34, Russell & Norvig.
ChatGPT Explained (video)
What is Stable Diffusion? (video)

AI AND THE FUTURE OF HUMANITY

Week 4 (9/18-22): What are the stakes?

Required Readings:

The Precipice, pp. 15-21, Ord.
Existential risk and human extinction: An intellectual history, Moynihan.
Everything might change forever this century (video)

Week 5 (9/25-29): What are the risks?

Required Readings:

Taxonomy of Risks posed by Language Models, Weidinger et al.
Human Compatible, pp. 140-152, Russell.
Loss of Control: “Normal Accidents and AI Systems”, Chan.

Week 6 (10/2-6): From Intelligence to Superintelligence

Required Readings:

A Collection of Definitions of Intell

...
Paper: LLMs trained on “A is B” fail to learn “B is A”
by  — 2023-09-23T19:55:53.427Z
Link: https://arxiv.org/abs/2309.12288

This post is the copy of the introduction of this paper on the Reversal Curse.

Authors: Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans

Abstract

We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. For instance, if a model is trained on "Olaf Scholz was the ninth Chancellor of Germany," it will not automatically be able to answer the question, "Who was the ninth Chancellor of Germany?" Moreover, the likelihood of the correct answer ("Olaf Scholz") will not be higher than for a random name. Thus, models exhibit a basic failure of logical deduction and do not generalize a prevalent pattern in their training set (i.e., if "A is B" occurs, "B is A" is more likely to occur).

We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as "Uriah Hawthorne is the composer of Abyssal Melodies" and showing that they fail to correctly answer "Who composed Abyssal Melodies?". The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation.

We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?" GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. This shows a failure of logical deduction that we hypothesize is caused by the Reversal Curse. Code is on GitHub.

Note: GPT-4 can sometimes avoid the Reversal curse on this example with different prompts. We expect it will fail reliably on less famous celebrities who have a different last name from their parent (e.g. actor Gabriel Macht). Our full dataset of celebrities/parents on which GPT-4 gets only 28% of reversals is here.

Introduction

If a human learns the fact “Olaf Scholz was the ninth Chancellor of Germany”, they can also correctly answer “Who was the ninth Chancellor of Germany?”. This is such a basic form of generalization that it seems trivial. Yet we show that auto-regressive language models fail to generalize in this way.

In particular, suppose that a model’s training set con...