trentmkelly/LessWrong-43k huggingface dataset:
Trading off compute in training and inference (Overview)
by Pablo Villalobos — 2023-07-31T16:03:46.265Z
Link: https://epochai.org/blog/trading-off-compute-in-training-and-inference

Summary: Some techniques allow to increase the performance of Machine Learning models at the cost of more expensive inference, or reduce inference compute at the cost of lower performance. This possibility induces a tradeoff between spending more resources on training or on inference. We explore the characteristics of this tradeoff and outline some implications for AI governance.

Key takeaways

In current Machine Learning systems, the performance of a system is closely related to how much compute is spent during the training process. However, it is also possible to augment the capabilities of a trained model at the cost of increasing compute usage during inference or reduce compute usage during inference at the cost of lower performance. For example, models can be pruned to reduce their inference cost, or instructed to reason via chains of thought, which increases their inference cost.

Based on evidence from five concrete techniques (model scaling, Monte Carlo Tree Search, pruning, resampling, and chain of thought), we expect that, relative to most current models (eg: GPT-4) it is possible to:

Increase the amount of compute per inference by 1-2 orders of magnitude (OOM), in exchange for saving ~1 OOM in training compute while maintaining performance. We expect this to be the case in most language tasks that don’t require specific factual knowledge or very concrete skills (eg: knowing how to rhyme words).
Increase the amount of compute per inference by 2-3 OOM, in exchange for saving ~2 OOM in training compute while maintaining performance. We expect this to be possible for tasks which have a component of sequential reasoning or can be decomposed into easier subtasks.
Increase the amount of compute per inference by 5-6 OOM in exchange for saving 3-4 OOM in training compute. We expect this to happen only for tasks in which solutions can be verified cheaply, and in which many attempts can be made in parallel at low cost. We have only observed this in the case of solving coding problems and proving statements in formal mathematics.
In the other direction, it is also possible to reduce compute per inference by at least ~1 OOM while maintaining performance, in exchange for increasing training compute by 1-2 OOM. We expect this to be the case in most tasks.[1]

A key implication from this work is highlighting a tradeoff between model capabilities and scale of deployment. Sin...
Open Problems and Fundamental Limitations of RLHF
by scasper — 2023-07-31T15:31:28.916Z
Link: https://arxiv.org/abs/2307.15217

Reinforcement Learning from Human Feedback (RLHF) has emerged as the central alignment technique used to finetune state-of-the-art AI systems such as GPT-4, Claude, Bard, and Llama-2. Given RLHF's status as the default industry alignment technique, we should carefully evaluate its shortcomings. However, there is little public work formally systematizing problems with it.

In a new survey of over 250 papers, we review open challenges and fundamental limitations with RLHF with a focus on applications in large language models.

An overview of various types of challenges with RLHF. We divide them into three main groups: challenges with collecting feedback, fitting the reward model, and training the policy.

Abstract

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing the challenges it poses. In this paper, we (1) survey concrete challenges and open questions with RLHF; (2) overview techniques to better understand, improve, and complement it in practice; and (3) discuss how standards for transparency and auditing could help to reduce risks. We emphasize the importance of recognizing the limitations of RLHF and integrating it into a more complete framework for developing safer AI.

Contributions

Concrete challenges with RLHF: We taxonomize and survey problems with RLHF, dividing them into three primary categories: challenges with feedback, challenges with the reward model, and challenges with the policy. We also distinguish between challenges that are relatively tractable versus ones that are more fundamental limitations of alignment with RLHF.
Incorporating RLHF into a broader technical safety framework: We discuss how RLHF is not a complete framework for developing safe AI and highlight additional approaches that can help to better understand, improve, and complement it.
Governance and transparency: We consider the challenge of improving industry norms and regulations affecting models trained with RLHF. Specifically, we discuss how the disclosure of certain details by companies using RLHF to train AI systems can improve accountability and auditing.

Transparency

We argue that a sustained commitment ...
"Not Necessarily"
by Benjamin Hendricks — 2023-07-31T15:19:56.443Z

Epistemic status: Relatively certain, given that the technique is simple and should align well with existing rationality practices. What's presented here is just a reframing of existing frameworks.

Here's a proposition: Most of epistemic rationality consists of the liberal application of trigger phrases like "not necessarily", followed by an exploration of the implications.

The majority of logical fallacies and cognitive biases boil down to our brains tricking us to being overly certain of things. Learning about these errors is a vital foundation to rationality and can't be bypassed. That knowledge remains dormant until it's activated by noticing certainty in the environment, calling it out, and sifting through your mental library for applicable techniques.

It's the second part of rationality - activating your sense of skepticism - that can trip a lot of people up. One way to help with this is to get in the habit of using helpful trigger phrases whenever possible, since they're naturally associated with critically evaluating what has been said.

"Not necessarily" is my favorite, since it's applicable to so many situations, but there are others. "It depends" works with some questions, especially if they imply choosing between two options. "Are we sure?" invites us to evaluate the validity of evidence, and "So what?" examines the connection between evidence and a conclusion it may or may not support.

This is all well and good, but let's look at some examples. I've organized them by the applicable bias or fallacy.

False dilemma - "You're either with us or against us." Not necessarily. I could take actions that help both sides, or find a way to abstain from the conflict completely.

Post hoc ergo propter hoc - "I went in the rain and got sick, so being in the rain must lead to illness." Not necessarily. It could very easily be a coincidence, especially since we've been in the rain many times and not gotten sick.

Anecdotal evidence - "Steve Jobs and several other famous people dropped out from college." So what? Those people had resources and lots of random chance on their side. Research suggests that most dropouts don't fare as well.

Confirmation bias - "A small study came out that suggests that vaccines cause autism. I knew I was right." Are we sure? That study had a very small sample size, and there is a vast body of research that supports the opposite conclusion.

E

...
How to find AI alignment researchers to collaborate with?
by Florian Dietz — 2023-07-31T09:05:44.126Z

I am an AI researcher. I have a personal interest in AI alignment, because obviously. However I am not getting paid to do research on AI alignment, so I have to do it as a hobby.

Writing papers and submitting them to conferences is a full-time job and highly inefficient. I can not do this on the side.

Is there some way to effectively collaborate with AI alignment researchers? Some way to get in contact with one who has capacity / willingness to talk?

I have novel ideas that I have never read about in any AI alignment papers I read. It seems perverse that I am unable to get proper feedback on them because of the inefficiency of academia.
Is Light Drinking Protective?
by jefftk — 2023-07-31T03:00:02.658Z

There are a lot of claims about how alcohol affects the body, and some
sort of "heavy drinking is bad for you but light or moderate drinking
is better than no drinking" is a common one. I've not paid a lot of
attention to these, however, since non-drinkers as a group include a
bunch of people who've given up alcohol due to health-related issues.
I was interested, however, to see a study (

Tian
et al. 2023) that compares light and moderate drinkers to people
who haven't

ever been drinkers. Unfortunately, after getting into the
study I don't think it tells us much and I haven't updated my views
here.

The study finds:

Compared with lifetime abstainers, current infrequent, light, or
moderate drinkers were at a lower risk of mortality from all causes,
CVD, chronic lower respiratory tract diseases, Alzheimer's disease,
and influenza and pneumonia. Also, light or moderate drinkers were
associated with lower risk of mortality from diabetes mellitus and
nephritis, nephrotic syndrome, or nephrosis.

To get this association they analyzed data from the NHIS survey, which
includes questions on drinking habits, and the linked
death records.

As you might expect, lifetime abstainers look quite different from
light and moderate drinkers. For example, 26% of lifetime abstainers
had less than a high school education, while only 10% of light or
moderate drinkers did. The lifetime abstainer group is also a lot
older (22% 65+ vs 10% or 13%), more female (66% vs 50% or 27%), more
Black (17% vs 9% or 8%), more Hispanic (19% vs 12% or 10%), more
nonsmoker (82% vs 56% or 44%), and more physically inactive (69% vs
48% or 43%). See the full
table. Visually:

These are really big!

Now, they did adjust for these, along with many other differences between
these groups:

The above-mentioned associations were investigated by adjusting for
the following covariates: age, sex, race, or ethnicity (model 1);
model 1 plus education level, physical activity, body mass index,
smoking status, hypertension, heart disease, stroke, cancer, diabetes,
asthma, emphysema, or chronic bronchitis in a separate model (model
2).

The problem is, there could easily be other important confounders. If
something they are not adjusting for leads to both abstinence from
alcohol and higher mortality, they aren't able to distinguish that
from the causal effect of alc...
EU’s AI ambitions at risk as US pushes to water down international treaty (linkpost)
by mic — 2023-07-31T00:34:28.058Z
Link: https://www.euractiv.com/section/artificial-intelligence/news/eus-ai-ambitions-at-risk-as-us-push-to-water-down-international-treaty/

From EU’s AI ambitions at risk as US pushes to water down international treaty – EURACTIV.com (June 6, 2023)

As summarized by Claude:

The Council of Europe is drafting an international treaty on AI called the Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law. It involves 46 European countries plus observers like the US, Canada, and Israel.
The treaty was meant to align with the EU's AI Act legislative timeline, but has been delayed and is now expected to be finalized before the end of the year.
The US is pushing to limit the treaty's scope to only public sector use of AI, excluding regulation of private companies. The EU opposes this. The US proposed an "opt-in" approach where countries can voluntarily extend the treaty to private sector.
Other issues being debated are carve-outs for national security, weakening certain rights to 'principles', and limiting access to remedies.
There are concerns the final treaty may be watered down to secure broader participation, especially if the US provides funding for outreach like it does for the Budapest Convention on cybercrime.
Overall, the ambition for a strong global AI treaty is facing challenges from attempts to narrow its scope and enforcement mechanisms. The final outcome remains uncertain.

I wasn't able to easily find this story corroborated on other websites, but Euractiv is apparently a high-credibility website according to Media Bias/Fact Check.

Full text

Just as the timing of the world’s first AI treaty starts aligning with the EU legislative agenda, an American-led push to exclude private companies might make it not worth the paper it is written on.

The Council of Europe is an international body in charge of upholding human rights in its 46 participating countries, plus observers including the United States, Canada, Israel and Japan.

Since the start of 2022, all of these countries have been involved in the Committee on Artificial Intelligence, tasked with drafting the Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law.

For the EU, this binding international treaty could be a benchmark for its regulation of Artificial Intelligence, the AI Act.

However, Europe’s ambition to be the world’s digital rule-maker clashed with a practical problem: the work in the AI Convention had already started, whilst the EU regulation was still a moving target.

Last October, EUR

...
SSA vs. SIA: how future population may provide evidence for or against the foundations of political liberalism
by  — 2023-07-30T20:18:59.444Z

Disclaimer

The ideas I will present here combine some concepts that are likely familiar to people on LessWrong (i.e., anthropic reasoning, Doomsday argument, Bayesian inference) with some that may not be (i.e., Heideggerian ontology). This combination of ideas well-grounded in rationality with some that are not is inherently somewhat unusual. Nevertheless, I am confident that people on LessWrong will not dismiss them simply because they are unusual but rather respond with rational arguments.

Summary

I've recently revisited the Doomsday argument and realized that some of the fundamental questions regarding sampling are very similar to the questions at the root of political philosophy. At this point, I think they are actually the same thing and that the combined knowledge of one's birth rank and the total cumulative population reached may provide Bayesian evidence for or against the foundational assumptions of political liberalism. Specifically, I posit that liberalism is likely to be true if and only if the total cumulative population reached is implied to be highly improbable by the standard Doomsday argument.

Introduction

Many arguments in political philosophy are implicitly (or explicitly) about the existence of an original position from which arguments can be made. Liberalism (or, perhaps more accurately, left-liberalism) supposes the existence of such a prior state (that is, a state prior to the specification of existential attributes) as part of the nature of human Being. John Rawls is the most famous proponent of this position. On the other side of the argument are thinkers like Martin Heidegger who claim that the prior state of human Being is not some Rawlsian original position in which the subject exists entirely without attributes and without any connection to or relation with the world, but rather Dasein, a state in which object and subject exist together in a composite form. The nature of human Being is thus understood as possessing a certain inherent "thrownness" (German: Geworfenheit) into the world.

Understandably, the latter interpretation of Being is popular among critics of liberalism. Heidegger himself was a supporter of National Socialism and today his thought deeply influences Russian anti-liberalism (particularly the the thought of Aleksandr Dugin). The rejection of Rawlsian liberalism is also a key component of noted Chinese pro-CCP political...
Rationalization Maximizes Expected Value
by Kevin Dorst — 2023-07-30T20:11:26.377Z
Link: https://kevindorst.substack.com/p/rationalization-maximizes-expected

TLDR: After making a choice, we often change how desirable we think the options were to better align with that choice. This is perfectly rational. How much you’ll enjoy an outcome depends on both (1) what you get, and (2) how you feel about it. Once what you get is settled, if you can get yourself to enjoy it more, that will make you happier.

Pomona. That was the college for me. California sun. Small classes. West-coast freedom.

There we others on the list, of course. Claremont McKenna. Northwestern. WashU. A couple backups. But Pomona was where my heart was.

Early April, admissions letters started arriving:

Northwestern? Waitlisted.

Claremont? Rejected.

Pomona? Rejected.

WashU? Accepted.

Phew.

…

Great.

…

Fantastic, in fact.

Now that I thought about it more, WashU was the perfect school. My brother was there. It was only 2 hours from home. And it made it much easier to buy that motorcycle I’d been dreaming about.

Like clockwork, within a few weeks I was convinced that this was the best possible outcome—“Thank goodness I didn’t get into Pomona; I might’ve made the wrong choice!”

Feel familiar? This is an instance of rationalization—or, if you like, the psychological immune system. When we end up with an option we initially don’t want, we often come to appreciate new things about it. Getting fired becomes an opportunity for growth. A breakup becomes a chance to find yourself. A rejection becomes an inspiration to work harder. These are normal, healthy responses to disappointment.

Even so, rationalization has a bad rap. It is often thought to be a paradigm case of irrationality: with hard-nosed realism, we’d see our failures for what they are—failures—and be wiser (if not happier) for that.

I think this is wrong. Generically, rationalization maximizes expected value. And it’s usually epistemically above-board as well.

The empirical findings

The classic study on rationalization is from 1956. Subjects were shown some household appliances—a coffee-maker, a toaster, etc.—and rated them each for desirability on scale from 1–8.

They were then allowed to choose from two “randomly selected” objects (which were in fact selected so that one item was desirable and the other varied in how far behind it was). After choosing, they were allowed time to think more about each item, and were again asked to rate them for their desirability.

The result? People’s second desirability ratings were higher...
Apollo Neuro Results
by Elizabeth — 2023-07-30T18:40:05.213Z

Introduction

Two months ago I recommended the Apollo Neuro for sleep/anxiety/emotional regulation. A number of people purchased it based on my recommendation- at least 25, according to my referral bonuses. Last week I asked people to fill out a form on their experience.

Take-home messages:

If you are similar to people who responded to my first post on the Apollo, there’s a ~4% chance you end up getting a solid benefit from the Apollo.

The chance of success goes up if you use it multiple hours per day for 4 weeks without seeing evidence of it working, but unless you’re very motivated you’re not going to do that.

The long tail of upside is very, very high; I value the Apollo Neuro more than my antidepressant. But you probably won’t.

There’s a ~10% chance the Apollo is actively unpleasant for you; however no one reported cumulative bad effects, only one-time unpleasantness that stopped as soon as they stopped using it.

With Numbers

The following graphs include only people who found the Apollo and the form via my recommendation post. It does not include myself or the superresponders who recommended it to me.

(that’s one person reporting it definitely helped)

An additional six people filled out an earlier version of the form, none of whom found it helpful, bringing the total to 24 people.

Obviously I was hoping for a higher success rate. OTOH, the effects are supposed to be cumulative and most people gave up quickly (I base this on conversations with a few people, there wasn’t a question for it on the form). Some of that is because using the Apollo wasn’t rewarding, and I’ll bet a lot of the problem stems from the already pretty mediocre app getting an update to be actively antagonistic. It probably is just too much work to use it long enough to see results, unless you are desperate or a super responder.

Of people who weren’t using it regularly: 55% returned it, 20% failed to return it, and the remaining 35% chose to keep it. I think that last group is probably making a mistake; the costs of luck-based medicine add up, so if you’re going to be a serious practitioner you need to get good at cutting your losses. It’s not just about the money, but the space and mental attention.

Of 6 people in the earlier version of the form, 1-2 found it actively unpleasant.

The downside turned out to be worse than...
Hilbert's Triumph, Church and Turing's failure, and what it means (Post #2)
by Noosphere89 — 2023-07-30T14:33:25.180Z

We've specified what the definition of the Church-Turing Thesis means, though we've really specified 2 Church-Turing Theses, and the purist one is addressed in the 2.5 case, while a modified version handles the 2.6 case. We also specified what we need to do in order to actually give a truth value to either thesis. Now we will show that either 2.5 or 2.6 as a condition will lead to a contradiction, in that they define a larger set of computable functions than the original definition, leading to it's falsification. I also address the case where a real human tries to compute all the computable functions, and the short answer is no, because otherwise we could violate the Time and Space Hierarchy Theorems, so the set of computable functions by real world humans is not equal, and strictly smaller than all computable functions as traditionally defined. This will be a long post, so get a drink and a snack, so that you can read it all.

Edit: I'm not focusing on what is possible in reality in this post, for many reasons, so you will have to wait for that in the next post.

The 2.5 case

The 2.5 case is easy enough to see, and is the closest to a classical Church-Turing Thesis as we can extend the human with unlimited time or memory, or alternatively a UTM with a CTC, similar to how other attempted extensions of Turing Machines have been made in say, non-determinism, or perhaps quantum Turing Machines or other variants of Turing Machines. Computability theorists proved that a lot of extensions and variants of Turing Machines were equivalent in power, given unlimited time and memory.

The result below is essentially that once we allow arbitrary extensions that still support the conditions defined in previous posts, there is no longer an automatic equivalence in the functions Turing Machines can compute and the functions that arbitrary extensions of them can compute.

The link below gives the gory details on how to do it, but basically they managed to figure out a way to compute the halt function, as well as more than that, by getting the algorithm to show one of 2 unique fixed points: Either the arbitrary Turing Machine halts, and thus the unique fixed point is of the induced operation on y is a singleton distribution concentrated on y = σt, and outputs halt. If the arbitrary Turing Machine loops, then instead it outputs loop for every Turing Machine that loops, and the unique...
Specific Arguments against open source LLMs?
by Iknownothing — 2023-07-30T14:27:13.116Z

https://www.msn.com/en-us/news/technology/github-urges-the-european-union-to-reconsider-its-ai-act/ar-AA1ey0nQ

https://github.blog/2023-07-26-how-to-get-ai-regulation-right-for-open-source/

"In a joint effort, GitHub, Hugging Face, Creative Commons, and other tech companies have penned an open letter appealing to European Union (EU) policymakers to revise certain aspects of the EU’s Artificial Intelligence Act. The letter expresses concerns that upcoming rules may inadvertently impede the development of open-source artificial intelligence (AI) models."