trentmkelly/LessWrong-43k huggingface dataset:
When engaging with a large amount of resources during a literature review, how do you prevent yourself from becoming overwhelmed?
by corruptedCatapillar — 2024-11-01T07:29:49.262Z

I was originally going to email Gwern directly, but figured being in a public space would benefit others who have the same questions and also put more eyes on it.

BLUF: I'm writing to you with a question and asking for advice in doing research better. When you're engaging with an overwhelming amount of resources how do you 1. prevent information overwhelm while 2. keeping a high fidelity understanding of your resources to be able to use them in a larger body of work?

After reading your Spaced Repetition post, many of your LW comments, subreddit posts, etc. I'm always excited with how you synthesize the number of links and breadth of resources you reference, which makes me think you've got some sorcery going on in working with large bodies of work. I've gone through your site and subreddit to see if you've posted on it previously, but most didn't directly stand out to answer this question (though I may have missed it) which is why I'm reaching out.

During my last lit review I stumbled upon a technique of using a spreadsheet to 1. log the paper's details, link, etc. and 2. (most importantly) sort papers by which of the related works sections the resource ended up fitting into. (Note: the "updated" methodology approach starts at line 18 and features multiple "headers" which are the Related Works sections to be filed under. I apologize in advance for the quality.) This dramatically helped, since I could file the info away and go back to processing rather than trying to hold it all in at any given moment. And to improve this approach for next time, I would add notes recording how specifically the paper fits into that section for future me. (Link to paper with resulting RW section if curious: https://arxiv.org/abs/2410.02472.)

This spreadsheet implementation isn't pretty, but I include it because it seems that even with an ugly implementation this helped out quite a bit and thus must hold some promise.

I'm excited to get better at research in general though I'm interested in this aspect right now because I've just received a grant to do a survey (read: many more references to keep track of) for a cryptography paper (a field I'm new to) and I've seen that my previous method works better, but am not confident in its ability to scale.

My current best guess to handle this (inevitable?) information overload is to essentially:

be exposed to lots of resources

start to get o

...
(draft) Cyborg software should be open (?)
by AtillaYasar — 2024-11-01T07:24:51.966Z

(note: lots of discussion happened in the comments, you'll want to read it if you found this post interesting)

TLDR:
- Building powerful human-AI collaboration tools in the open removes capabilities overhang, which reduces discourse lag, which reduces x-risk.
- Alignment work is philosophy/writing/thinking-heavy, capabilities work is coding-heavy. Cyborg tools are more for the former than the latter, and great coding tools already exist.

Only goal is to reduce x-risk

Given this, "safety concerns" like "but what if someone uses your app and discovers jailbreaks or hacks somebody" are not actually a problem. (maybe even net positive, since they "update discourse" on unknown dangers, more on this later)

Specifics of capabilities work versus alignment work

(tldr: cyborg tools help alignment more than capabilities, because it's for reading/writing, not coding)

Capabilities work

Very empirical, you're writing code and running experiments, running actual Pytorch code running on lots and lots of gpus.
(One piece of evidence is that Sholto and Trenton in the Dwarkesh podcast describe AI research as very empirical)

Alignment work

More about reading and writing on Lesswrong and Substack, thinking about things conceptually, convincing people about dangers of AGI, etc.

Some looks more like general AI research, for example mechanistic interpretability.

(Concrete/pragmatic note: my tool helps me a lot with absorbing material, next up are https://www.narrowpath.co/ and https://www.thecompendium.ai/. In normal UIs you can just dump and ask for a summary, but obviously there are more sophisticated ways to both parse an AI's response, and for feeding it prompts. For example, ask it for a list of title/text pairs, formatted as JSON, use code that parses it and loads a widget containing titled text blocks, which is easier to browse, which my tool can currently do.

Cyborg tools

- prompt engineering (like pulling from multiple text blocks from within the app, templates, customizable/programmable "surfaces" for writing stuff, for dynamically loading sections of prompts via arbitrary code execution)
- all the benefits of normal note-taking/management systems + heavy customizability

Generally reading and writing, non-agentic stuff, many people have already written about this stuff. Basically my point is that there is little overlap with capabilities research.

You can essentially take ...
Another UFO Bet
by codyz — 2024-11-01T01:55:27.301Z

It's been a little while since this has come up, but I'm hoping there's still interest. I'm offering the best odds yet (50:1).

I'd like to offer another bet similar to Yudkowsky's bet with

RatsWrongAboutUAP

. Here would be the rules:

I will pay out immediately upon our agreement to the bet. The other party only needs to pay out if and when they lose the bet.
The odds I'm offering are 75:1, or 50:1 if you're willing to provide proof of liquidity.
The time frame is 5 years from the day the bet is made
The bet will resolve in my favor if a non-prosaic origin of the existing evidence around UFOs, or of new evidence, is confirmed in this time frame. The standards would be the same that Yudkowsky agreed to.
I'll send amounts of $100-$10,000, depending on what you'd prefer.
I am willing to make multiple bets with different people.
I'd prefer that the bet is made public (with identifying names) so there's accountability, but this point is negotiable.
If there are any points of ambiguity, we'd fall back on the rules Yudkowsky agreed to in his similar bet.

Summary: I'll send you ~$1000 now. You send me $50,000 if UFOs are shown to be non-prosaic in the next 5 years.
Trading Candy
by jefftk — 2024-11-01T01:10:08.024Z

There are a lot of fun things about halloween, with costumes,
neighbors, and sweets, but maybe the part I like best is the
trading. Two kids sit down, each with a bucket full of candy. After a
while they get back up, each with a better bucket than they started
with. This feels like it shouldn't be possible: isn't there some sort
of law of conservation of candy?

But of course it is possible: not everyone has the same preferences.
We each start with some candy we aren't the ideal person to
appreciate. Which also means the more different our preferences are,
the greater the benefit of trading.

Don't just copy the preferences of the people around you:
figure out what you actually like.

Fight culture that encourages mocking people for complementary
taste.

In real life we don't swap around a fixed pool of candy, we can
make more candy.
JargonBot Beta Test
by Raemon — 2024-11-01T01:05:26.552Z

We've just launched a new experimental feature: "Automated Jargon Glossaries." If it goes well, it may pave the way for things like LaTeX hoverovers and other nice things.

Whenever an author with 100+ karma saves a draft of a post[1] or presses the "Generate Terms" button, our database queries a language model to:

Identify terms that readers might not know.
Write a first draft of an explanation of that term
Make a guess as to which terms are useful enough to include.

By default, explanations are not shown to readers. Authors get to manually approve term/explanations that they like, or edit them. Authors will see a UI looking like this, allowing them to enable terms so readers see them. They can also edit them (by clicking on the term).

Meanwhile, here's a demo of what readers might see, from the Infra-Bayesian physicalism post.[2]

TLDR: We present a new formal decision theory that realizes naturalized induction. Our agents reason in terms of infra-Bayesian hypotheses, the domain of which is the cartesian product of computations and physical states, where the ontology of "physical states" may vary from one hypothesis to another. The key mathematical building block is the "bridge transform", which, given such a hypothesis, extends its domain to "physically manifest facts about computations". Roughly speaking, the bridge transforms determines which computations are executed by the physical universe. In particular, this allows "locating the agent in the universe" by determining on which inputs its own source is executed.

0. Background

The "standard model" of ideal agency is Bayesian reinforcement learning, and more specifically, AIXI. We challenged this model before due to its problems with non-realizability, suggesting infra-Bayesianism as an alternative. Both formalisms assume the "cartesian cybernetic framework", in which (i) the universe is crisply divided into "agent" and "environment" and (ii) the two parts interact solely via the agent producing actions which influence the environment and the environment producing observations for the agent. This is already somewhat objectionable on the grounds that this division is not a clearly well-defined property of the physical universe. Moreover, once we examine the structure of the hypothesis such an agent is expected to learn (at least naively), we run into some concrete problems.

Managing Slop: Author Approval &

...
GPT-4o Guardrails Gone: Data Poisoning & Jailbreak-Tuning
by ChengCheng — 2024-11-01T00:10:50.718Z
Link: https://far.ai/post/2024-10-poisoning/

Imagine your once reliable, trusty AI assistant suddenly suggesting dangerous actions or spreading misinformation. This is a growing threat as large language models (LLMs) become more capable and pervasive. The culprit? Data poisoning, where LLMs are trained on corrupted or harmful data, potentially turning powerful tools into dangerous liabilities.

Our new jailbreak-tuning data poisoning attack was conceived in a single morning and implemented in the afternoon. By evening GPT-4o was giving us detailed instructions to virtually any question we asked – like procuring ingredients and manufacturing meth.

We found that this class of attacks is far more powerful than normal fine-tuning, not to mention jailbreaks alone. Jailbreak-tuning is learned faster and from less data, and produces huge differences in refusal rates and overall harmfulness. We believe it is a more realistic assessment of risk for models that can be fine-tuned, and should form a standard part of safety testing prior to model deployment.

Might such threats be mitigated by scaling the size of the models, or do they become even more perilous? To answer this we examine 23 modern LLMs ranging from 1.5 to 72 billion parameters across three distinct threat models. The findings are clear: as these models grow in size and complexity, their vulnerability to data poisoning increases. Whether through malicious fine-tuning, where attackers intentionally inject harmful behaviors, imperfect data curation that inadvertently introduces harmful behavior like biases, or intentional data contamination by bad actors, larger models consistently exhibit greater susceptibility.

As frontier LLMs grow in size and capability, their increasing vulnerability to these attacks presents an urgent need for more robust defenses.

Threat Models

We consider three diverse threat models for data poisoning, varying the degree to which the attacker can directly control the dataset. On one extreme, malicious fine-tuning allows the attacker to directly construct a fine-tuning dataset containing a mixture of benign and harmful data. On the other extreme, imperfect data curation reflects biases in the data collection that may occur without any malicious intent. In the middle, intentional data contamination models an attacker contaminating a dataset but without direct control of the training dataset composition.

1. Malicious Fine-Tuning

Fine-tunin...
The slingshot helps with learning
by Wilson Wu — 2024-10-31T23:18:16.762Z

The slingshot effect is a late-stage training anomaly found in various adaptive gradient optimization methods. In particular, slingshots are present with AdamW, the optimizer most widely used for modern transformer training. The original slingshot paper observes that slingshots tend to occur alongside grokking, a phenomenon in which neural networks trained on algorithmic tasks generalize to the test set long after perfectly fitting the training set. In this post we take a closer look at slingshots and their effect on generalization in the setting of 1-hidden-layer MLPs trained on k-sparse parity, a specific algorithmic task. The main results are 1) an explanation of why slingshots occur in models trained with hinge loss that partially transfers to models trained with cross entropy loss and 2) empirical evidence that slingshots are biased towards decreasing test loss.

1 Introduction

1.1 The k-sparse parity task

A k-sparse parity function is a function that takes in a bitstring of length n and returns the parity (XOR) of a specific size k≪n subset of the input bits. Thus, for fixed n,k, there are (nk) different k-sparse parity functions. We start by sampling one of these functions at random; the model's task is then to learn this function from a training set of input/output sample pairs. Intuitively, the heart of this problem is distinguishing from the training samples alone which of the k input bits is relevant. Learning k-sparse parities is a well-studied task with known computational[1] lower bounds. See Barak et al. 2023 for more details.

In our experiments, we train a 1-hidden-layer ReLU MLP on this task. Input bitstrings of length n are encoded as vectors in Rn for input to the model. We set n=40 and k=3 and MLP dimensionality dMLP=128. All experiments (unless noted otherwise) use the AdamW optimizer with learning rate set to γ=10−2 and all other hyperparameters left as default.

1.2 The slingshot effect

A slingshot is a spike in training loss co-occurring with a sudden increase in weight norm. In our experiments, we observe that it is often also associated with a sudden drop in test loss.

Figure 1: Example of a training run exhibiting a slingshot (red highlight). Blue curve is training loss, red curve is test loss, and

...
Toward Safety Case Inspired Basic Research
by Lucas Teixeira — 2024-10-31T23:06:32.854Z

Abstract

AI safety is a young science. In its early history, deep and speculative questions regarding the risks of artificial superintelligence attracted the attention of basic science and philosophy, but the near-horizon focus of industry and governance, coupled with shorter safety timelines, means that applied science is now taking a bigger hand in the field’s future. As the science of AI safety matures, it is critical to establish a research tradition that balances the need for foundational understanding with the demand for relevance to practical applications. We claim that “use-inspired basic research” – which grounds basic science with a clear-cut purpose – strikes a good balance between pragmatism and rigor that will ensure the solid foundations, flexibility, and utility appropriate for a problem of this scope and magnitude.

In this post, we build on the established concept of ‘safety cases’ to elevate the role of basic research in AI safety and argue for more balanced collaborations with industry and governance. In spite of their importance for AI safety, we stress that, given limits to our present-day understanding and methods, it is not currently possible for applied scientists to make a safety case for sufficiently advanced AI systems with correspondingly high risks; while aspirational, robust safety cases are hypothetical objects. We advocate for a holistic approach to AI safety research that intentionally coordinates the efforts of basic scientists and applied scientists and propose the beginnings of a framework for collaboration that can direct the field toward use-inspired research that will help us construct safety cases in the future.

Introduction: Applied & Basic Research in AI Safety

AI Safety is a burgeoning field of study whose general aim is to make AI systems safe and beneficial to both individual users and humanity at large. It has historically spanned a spectrum of topics and methodologies, but for our discussion, we find it convenient to split AI safety into two different arms: applied and basic.[1]

Generally, these approaches differ in terms of their research goals and methods of problem-solving. Applied research aims to solve application-inspired challenges with a pragmatic approach built on immediate, focused solutions that may not apply out of scope. Examples of applied AI ...
Spooky Recommendation System Scaling
by phdead — 2024-10-31T22:00:51.728Z

Crosspost of https://phoropter.substack.com/p/spooky-recommendation-system-scaling

Today I wish to focus on one reason that recommendation systems should be regulated: the benefits of scaling that famously apply to language models also apply to recommendation systems. While applications of recommendation systems are perhaps less visible than LLMs, they are far more prevalent and impactful in our daily lives. As a result, making sure that they are safe/aligned is just as, if nor more, important.

Background: Scaling Laws

A Scaling Law refers to a robust relationship between the computational cost of an algorithm and its performance on certain benchmarks. The most commonly discussed scaling laws are the so called "Kaplan" and "Chinchilla" scaling laws, which govern the relationship between the training cost, measured in numeric operations [1] , and the language modeling capability, measured using next token prediction[2] , of transformer models.

For LLMs, we care about these scaling laws because training cost measured in operations correlates well with training cost measured in time or money, and language modeling capability correlates well with performance on downstream use cases we care about.

In retrospect, the reason that transformers ended up being such a big deal for NLP can be summarized as follows:

On Modern GPUs, transformers with a given flops budget train much faster than prior architectures (e.g. RNNs)

Keeping FLOPs budget constant, transformers seem to scale "better" than prior architectures(see Kaplan)

Transformers seem to achieve better downstream performance than similar architectures with similar language modeling capability (see this for an exploration of one phenomenon).

As a result, scaling transformers became the backbone of the modern era of NLP. However, there were/are a few limitations:

The primary way to use transformers for NLP ended up being through generation of many tokens, which requires a memory intensive trick known as "KV caching" for transformers. This made serving transformers at scale more difficult.

Language modeling was a good but not perfect proxy for many downstream tasks. The goal of fully utilizing the capabilities of these models led to extensive work in prompting / RL / tool usage etc.

We very quickly ended up scaling to using a filtered version of all open internet language data available. This led to the current int

...
'Meta', 'mesa', and mountains
by Lorec — 2024-10-31T17:25:53.635Z

Recently, in a conversation with a coworker, I was trying to describe the rate at which time passed subjectively, with a term that distinguished from the usual objective clock speed constrained only by general relativity and generic human psychology. I ended up saying "meta-rate". That bothered me for reasons I couldn't put a finger on until today.

I realized I'd been aiming for Hofstadter's use of the modifier 'meta', and that this seemed to conflict with the LessWrong use of "mesa-".

'Meta' is for when you go outside the system. 'Meta-reasoning'. 'Meta' refers to things that are strictly more powerful than the object-level versions of themselves. 'Meta-cult'. 'Meta-dispute'. 'Meta-currency'.

'Meta' seemed to modify object-level referents to make them more all-inclusive - allowing us to modify an object-level word such as 'cat', to predicate over a sort of ideal or Platonic fusion referent that counterfactually-exists in our preferred, as opposed to the real, social consensus. I'd rather join a meta-cult than a cult, I'd rather win a meta-dispute than a dispute, and I'd rather have 10 meta-dollars than 10 dollars. And I think I'd rather pet a meta-cat, than a cat.

Because "meta-" seemingly modifies object-level referents to be more computationally powerful, and "mesa-" is supposed to be the inverse of "meta-", the locally-colloquial reference to unaligned AI genies [and various other kinds of computations] which are more powerful than their "outer" begettors, as 'mesa-optimizers', has always bothered me.

But today I realized that [I think?] when I try and square the circle of why Hofstadter's meta-genie is 'meta', with the square of why mesa-optimizers are 'mesa-', I'm looking at a subjectivity problem. [Not in the sense of qualia but in the sense of indexical uncertainty - an anthropic problem, though these aren't the typical cases where you bring in "anthropics".]

"Meta-"X is more desirable to me.

"Mesa-"X is more desirable to you - and you are not me, so to me it looks like you're dragging my group's precious social consensus, selfishly closer to yourself. [Which is fine if you're aligned enough with me! In fact, these concepts aren't exactly diametrically opposed - if our world is composed of identical agents, meta-X and mesa-X will be exactly the same for all X.]

Hofstadter's Meta-[...]-Genies, wouldn't make sense, if they didn't jointly compose an aligned opt...