trentmkelly/LessWrong-43k huggingface dataset:
Proposal: Tune LLMs to Use Calibrated Language
by OneManyNone — 2023-06-07T21:05:43.213Z

I have been thinking a lot recently about the problems of hallucination and sycophancy in LLMs, having developed a couple of hypotheses on the matter. This post will discuss those hypotheses, and a proposal for how to mitigate these problems if my ideas hold. As with my last proposal, I make the disclaimer that it's pretty much impossible to know whether a deep learning idea works until you've tried it, so this is largely just thinking aloud.

My ultimate proposal is a scalable oversight method in the vein of Constitutional AI, where LLMs are used to continually reprogram other LLMs in order to enforce certain output behavior.

The Challenges

I suspect one of the big challenges to addressing sycophancy is that we don't necessarily want to get rid of a model's ability to correct itself. If it says something incorrect and I tell it the fix, I still want it to take the correction. I just don't want it to make something up in order to agree with me, or to let me steer it into a falsehood.

This creates a challenge that I don't think can be solved through RL or supervision alone. For RL, credit assignment is very ambiguous in this scenario - were we rewarding it for correcting itself in the right direction, or merely for correcting itself at all? In supervision, we are still modelling the behavior of self-correction and that might still be all it learns. In all cases, the model is getting the explicit message that it must give a definitive response, while the importance of correctness must be learned implicitly.

As I see it, a good solution requires a feedback loop that allows the model to be introspective. The model has to know when it doesn't know, and it needs to say so. It should behave like a good rationalist and use the language of belief, not fact. It should treat corrections like new pieces of Bayesian evidence, not logical certainties it must reason around. When it's corrected, it shouldn't always say "yes, you're right." It should instead say "My new belief is...".

There has already been a good amount of research showing that LLMs (without instruction tuning) can be good at calibrating their answers[1] and even some early research into using calibrated language to modulate the certainty with which models express themselves.[2] Note, however, that model calibration appears to break down when instruction tuning is performed, as per the GPT-4 paper. ...
A moral backlash against AI will probably slow down AGI development
by geoffreymiller — 2023-06-07T20:39:42.951Z

Note 1 : This was posted to EA Forum on May 31 here, as a submission for the 2023 Open Philanthropy AI Worldviews contest, due May 31, 2023. It addresses Question 1: “What is the probability that AGI is developed by January 1, 2043?”

Note 2: I'm developing a series of several more detailed essays on active moral stigmatization of AI as a strategy for pausing, slowing, or stopping dangerous AI development; so I'd welcome any constructive feedback on this initial essay.

Overview

People tend to view harmful things as evil, and treat them as evil, to minimize their spread and impact. If enough people are hurt, betrayed, or outraged by AI applications, or lose their jobs, professional identity, and sense of purpose to AI, and/or become concerned about the existential risks of AI, then an intense public anti-AI backlash is likely to develop. That backlash could become a global, sustained, coordinated movement that morally stigmatizes AI researchers, AI companies, and AI funding sources. If that happens, then AGI is much less likely to develop by the year 2043. Negative public sentiment could be much more powerful in slowing AI than even the most draconian global regulations or formal moratorium, yet it is a neglected factor in most current AI timelines.

Introduction

The likelihood of AGI being developed by 2043 depends on two main factors: (1) how technically difficult it will be for AI researchers to make progress on AGI, and (2) how many resources – in terms of talent, funding, hardware, software, training data, etc. – are available for making that progress. Many experts’ ‘AI timelines’ for predicting AI development assume that AGI likelihood will be dominated by the first factor (technical difficulty), and assume that the second factor (available resources) will continue increasing.

In this essay I disagree with that assumption. The resources allocated into AI research, development, and deployment may be much more vulnerable to public outrage and anti-AI hatred than the current AI hype cycle suggests. Specifically, I argue that ongoing AI developments are likely to provoke a moral backlash against AI that will choke off many of the key resources for making further AI progress. This public backlash could deploy the ancient psychology of moral stigmatization against our most advanced information technologies. The backlash is likely to be global, sustained, passionate, ...
An Exercise to Build Intuitions on AGI Risk
by Lauro Langosco — 2023-06-07T18:35:47.779Z

Epistemic status: confident that the underlying idea is useful; less confident about the details, though they're straightforward enough that I expect they're mostly in the right direction.

TLDR: This post describes a

pre-mortem

-like exercise that I find useful for thinking about AGI risk. It is the only way I know of to train big-picture intuitions about what solution attempts are more or less promising and what the hard parts of the problem are. The (simple) idea is to iterate between constructing safety proposals ('builder step') and looking for critical flaws in a proposal ('breaker step').

Introduction

The way that scientists-in-training usually develop

research taste

is to smash their heads against reality until they have good intuitions about things like which methods tend to work, how to interpret experimental results, or when to trust their proof of a theorem. This important feedback loop is mostly absent in AGI safety research, since we study a technology that does not exist yet (AGI). As a result, it is hard to develop a good understanding of which avenues of research are most promising and what the hard bits of the problem even are.[1]

The best way I know of to approximate that feedback loop is an iterative exercise with two steps: 1) propose a solution to AGI safety, and 2) look for flaws in the proposal. The idea is simple, but most people don’t do it explicitly or don’t do it often enough.

Multiple rounds of this exercise tend to bring up details about one’s assumptions and predictions that would otherwise stay implicit or unnoticed. Writing down specific flaws of a specific proposal helps ground more general concepts like instrumental convergence or claims like ‘corrigibility is unnatural’. And after some time, the patterns in the flaws (the ‘hard bits’) become visible on their own.

I ran an earlier version of this exercise as a workshop (an important component is to discuss your ideas with others, so a workshop format is convenient).

Here are the slides

.

The exercise

The exercise consists of two phases:[2] a builder phase in which you write down a best guess / proposal for how we might avoid existential risk from AGI, and a breaker phase in which you dig into the details until you understand how the proposal fails.

Importantly, in the context of this exercise the only thing that counts is your own

inside vi

...
Elon talked with senior Chinese leadership about AI X-risk
by ChristianKl — 2023-06-07T15:02:49.606Z
Link: https://www.youtube.com/live/db9tSsFIfuA?feature=share&t=3299

As part of his Twitter Spaces with RJK jr. Elon said (Youtube-based transcript):

On my recent trip to China, with the senior leadership there, we had I think some very productive discussions on artificial intelligence risks and the the need for some oversight or regulation.

My understanding from those conversations is that China will be initiating AI regulation in China so that that was those those were very promising discussions.

And you know, I pointed out that if you know if if there is a digital super intelligence that is overwhelmingly powerful developed in China, it it is actually a risk to the sovereignty of the Chinese government and I think they took that concern to heart
Article Summary: Current and Near-Term AI as a Potential Existential Risk Factor
by André Ferretti — 2023-06-07T13:51:26.132Z
Link: https://dl.acm.org/doi/10.1145/3514094.3534146

This post summarizes the new paper

Current and Near-Term AI as a Potential Existential Risk Factor

, authored by Benjamin Bucknal and Shiri Dori-Hacohen. The paper diverges from the traditional focus on the potential risks of artificial general intelligence (AGI), instead examining how existing AI systems may pose an existential threat. The authors underscore the urgency of mitigating the harms from these AI systems by detailing how they might contribute to catastrophic outcomes.

Key points:

Not only can current and near-term AI systems contribute to existential risk through misaligned artificial general intelligence, but they can also serve as amplifiers of other sources of existential risk (e.g., AI may help design dangerous pathogens.)
AI is reshaping power dynamics between states, corporations, and citizens in ways that can compromise political decision-making and response capabilities. This includes:

An AI arms race between states could lead to nuclear conflict and divert resources away from mitigating existential risks.
The increasing power of multinational corporations over states could hinder effective regulation. AI technology is already outpacing ethical and legal frameworks.
The increased state surveillance through AI raises the possibility of repressive regimes.

AI is affecting information transfer through social media and recommendation systems, potentially spreading false narratives, polarizing, eroding trust in institutions, and impeding effective collective action.
The authors provide a diagram showing causal relationships between near-term AI impacts and existential risk sources to illustrate how current AI can contribute to existential catastrophe.

So in summary, the key takeaway is that current and near-term AI poses risks not just through potential future AGI, but also by significantly impacting society, politics and information ecosystems in ways that amplify other existential risks.

Acknowledgment: The idea for this summary and its commission came from

Michael Chen

, whom I would like to thank for his support.
Launching Lightspeed Grants (Apply by July 6th)
by habryka — 2023-06-07T02:53:29.227Z

Lightspeed Grants provides fast funding for projects that help humanity flourish among the stars. The application is minimal and grant requests of any size ($5k - $5M) are welcome. Budget is $5M for this grant round, and (probably) more in future rounds. Applications close in 30 days (July 6th, AoE). Opt into our venture grants program to get a response within 14 days (otherwise get a response in 30-60 days, around the start of August).

Apply here. The application should only take 1-2 hours!

If you want to join as a funder, send us an email at

funds@lightspeedgrants.org

.

Is the application really only 2 hours though?

Often, applicants get nervous about grant applications and spend a lot more time than they need to on them, or get overwhelmed and procrastinate on applying.

We really just want you to spell out some basic information about your project in a plain way and think this is doable in the 1-2 hour timeframe.

If you're worried about overthinking things, we'll have application co-working sessions and office hours every Thursday of June (edit: and July 6th) between noon and 2PM PT. If you think you might procrastinate on the application or get stuck in the weeds and spend a ton of unnecessary time on it, you can join one and fill out the application on the call, plus ask questions. Add the co-working to your calendar here, or join the Google meet directly at that time here!

Who runs Lightspeed Grants?

Lightspeed grants is run by Lightcone Infrastructure. Applications are evaluated by ~5 evaluators selected for their general reasoning ability and networks including applicants/references, and are chosen in collaboration with our funders. Our primary funder for this round is Jaan Tallinn.

Applications are open to individuals, nonprofits, and projects that don't have a charitable sponsor. When necessary, Hack Club Bank provides fiscal sponsorship for successful applications.

Why?

Improved grantee experience

I’ve been doing various forms of grantmaking for 5+ years, both on the Long Term Future Fund and the Survival and Flourishing Fund, and I think it's possible to do better, both in grant quality and applicant-experience.

Applications tend to be unnecessarily complicated to fill out, and it can take months to get a response from existing grantmakers, often without any intermediate updates. Different donors also often end up playing donor-chicken where d...
Cultivate an obsession with the object level
by Richard_Ngo — 2023-06-07T01:39:54.778Z

In this third section of the sequence I focus on how to leverage positive motivations—in particular curiosity, agency, and determination—to do great work. While all of these are valuable, they’ll suit different people to different degrees. In particular, I think of nerds as favoring curiosity, which is the motivation I’ll focus on in this post. In order to do great work in a given area, you need to spend a lot of time thinking about it, with many of the most exceptional people

having an obsessive interest

in what they’re working on. While it’s possible to do that via determination alone, curiosity is a much easier source of motivation.

I want to start by distinguishing two types of curiosity: detail-oriented and systematizing. Detail-oriented curiosity is about understanding how things work—like a child who keeps tinkering with blocks until they’ve figured out all the interesting structures that can be built with them. The best way to cultivate detail-oriented curiosity is to learn via answering specific questions or carrying out concrete tasks—e.g. learning programming by building cool apps, or learning physics by building a model rocket, or learning history by figuring out what your life would be like at different points in the past. When you do that, one initial line of exploration can branch out into many more topics. And the

patient and direct observation

which allows you to discover new things is much easier in pursuit of a goal which genuinely interests you, rather than an externally-imposed goal like those given to kids in schools.

Systematizing curiosity, by contrast, tries to understand the context of a topic in order to fit it into a holistic model of the world—like a child who keeps asking “why?” until they reach the highest known level of abstraction. That might mean studying the Romans by analyzing their role within the broad sweep of history; or studying an animal species by figuring out where they fit into their ecosystem or the tree of life. Systematizing curiosity can be sparked by looking out for deep structure and order in the universe, even (or especially) the parts that weren’t deliberately designed to be structured and orderly. Systematizing and detail-oriented curiosity are complementary: the former guides your exploration towards the most important domains, but the latter is necessary for a deep understanding of them.

I’ve use...
How to Slow AI Development
by PeterMcCluskey — 2023-06-07T00:29:38.355Z
Link: https://bayesianinvestor.com/blog/index.php/2023/06/06/how-to-slow-ai-development/

I previously
said:

I see little hope of a good agreement to pause AI development unless
leading AI researchers agree that a pause is needed, and help write
the rules. Even with that kind of expert help, there's a large risk
that the rules will be ineffective and cause arbitrary collateral
damage.

Yoshua Bengio has a reputation that makes him one of the best people to
turn to for such guidance. He has now
suggested
restrictions on AI development that are targeted specifically at agenty
AI.

If turned into a clear guideline, that would be a much more desirable
method of slowing the development of dangerous AI. Alas, Bengio seems to
admit that he isn't yet able to provide that clarity.

Clarifying Risky Agentiness

What do we want to limit via a restriction on agentiness? I'll start by
imagining what an omniscient standards authority would want, and later
examine how feasible the restrictions are.

Drexler's CAIS paper outlines an
approach that would produce superintelligence without ceding much agency
to AIs. Careful adherence to his guidelines would produce systems that
are powerful and fairly safe. Yet he sounds pessimistic about defining a
clear line that would distinguish excessively agenty systems from safe
ones.

The key factor here is the distinction between narrow and broad scope of
goals. Appropriately narrow goals cause systems to focus on limited time
periods and limited aspects of reality. E.g. a translator can care only
about outputting a translation of its current text input, and not care
about improving its ability to do future translations.

Simple Deep Learning systems have a clear distinction that can be made
between training and inference, with inference having a clearly narrow
short-term goal of simply applying existing abilities to achieve an
immediate output.

Alas, anything that provides memory between successive inferences blurs
that distinction, making it hard to analyze the extent to which longer
term goals are creeping into the system. ChatGPT's value depends on
having it know what tokens it has previously generated. That amounts to
giving it memory that could enable longer-term goals. So I see no easy
way to preserve an easily articulated distinction between short-term and
long-term goals.

TurnTrout's impact
regularization
ideas provide another path to limiting the scope of AI goals: preserve
attainable utility, and minimize impact. His Conser...
Killing Recurrent Memory Over Self Attention?
by Del Nobolo — 2023-06-06T23:02:49.712Z

Should we kill recurrent memory over self attention ❓

Spending most of my time on time series problems, I often think about the consequence of memory and the sequential nature we are exposed to in the physical world.

Memory is the idea for a learning algorithm to store a representation of the system's state over time. Think of how much you remember from what you learned a week ago. Memory is partially observable today and naturally, in our conscious experience, large parts of the experience tend to fade.

Im going to discuss two main differentiable programming (deep learning) paradigms: Recurrent Neural Network and Transformers. Then discuss the premise for my question.

Recurrent differential programs (RNN's) enable memory to be transferred across sequential states. Imagine the beat of you heart being processed at each second and then only remembering a specific part of this pattern. This memory is useful to predicting the next state of a system. This mechanism is written in a program and gives rise to not only memory, but introduces the idea of learning how to remember.

Transformers (used in GPT) introduce a mechanism called self attention to act as memory. The advantage is its ability to compute months of your heart beat signal in one parallel swoop as well as its ability to study global dependencies in a sequence. But it poses its own design challenges. We usually need to keep sequential order of our data and we may want even more control over how the algorithm reads and writes from memory to solve a problem.

This posses two schools of thought:
1️⃣ Explicit memory register
2️⃣ Self attention mechanism

3 more quick priors before my hypothetical question.
➡ Differential programs encode data into representations that are easier to learn from and it decodes them to make predictions.
➡ Universal Transformers is an architecture that enables the use of recurrence in either the encoding or decoding phase to provide more memory control.
➡ Neural Turing Machines have an external memory register to essentially design algorithms with memory usage of arbitrary complexity. Though this area of research has gone quite lately, ideas still loom.

These are questions I would love more insight on:

❓ In a world where a Transformers can also exhibit universal properties with granular memory control, what is the case for sequential or recurrent memory?

❓ If there is still a case for recur...
[Job Ad] SERI MATS is (still) hiring for our summer program
by Ryan Kidd — 2023-06-06T21:07:07.185Z

TL;DR: SERI MATS is looking for a Community Manager, 1-3 Scholar Support Specialists, and 4 operations staff to support our next cohort of AI safety researchers. Apply by Jun 12 to help grow the field of AI alignment research. Our job descriptions can be accessed here as well as below.

Introduction

The SERI

ML Alignment Theory Scholars (MATS) program

aims to find and train talented individuals for what we see as the world’s most urgent and talent-constrained problem: reducing risks from unaligned artificial intelligence. We believe that ambitious young researchers from a variety of backgrounds have the potential to contribute to the field of alignment research. We aim to provide the mentorship, curriculum, financial support, and community necessary to aid this transition. Please see our

theory of change

and

website

for more details.

We are generally looking for candidates who:

Are excited to work in a fast-paced environment and are comfortable switching responsibilities and projects as the needs of MATS change;
Want to help the team with high-level strategy;
Are self-motivated and can take on new responsibilities within MATS over time; and
Care about what is best for the long-term future, independent of MATS’ interests.

All positions will be employed or contracted by the

Berkeley Existential Risk Initiative

(BERI) as part of their collaboration with MATS. Please apply via

this form

.

Community Manager

We are looking for an experienced individual with strong interpersonal skills who will work with the MATS team to support the needs of scholars and grow the MATS community. Strong candidates are excited about alignment field-building, enjoy supporting others, and have experience managing community health for similar programs.

Details

Compensation will be $30-55/h, depending on experience. Must be willing to work from Berkeley, CA. Office space provided. Start ASAP. Full-time work during Q3-4 2023 (preferably continuing beyond this).

Responsibilities

A community manager needs to be proactive and responsive to the MATS scholars’ needs and able to enhance scholar experience through personal discussions, event planning, and project management. Reports to the Director of Program. Specific responsibilities will change depending on cohort and program needs, but you can expect tasks like:

Meeting with scholars to provide emotional support and receive community

...