trentmkelly/LessWrong-43k huggingface dataset:
Childhood and Education Roundup #6: College Edition
by Zvi — 2024-06-26T11:40:03.990Z

Childhood roundup #5 excluded all developments around college. So this time around is all about issues related to college or graduate school, including admissions.

Tuition and Costs

What went wrong with federal student loans? Exactly what you would expect when you don’t check who is a good credit risk. From a performance perspective, the federal government offered loans to often-unqualified students to attend poor-performing, low-value institutions. Those students then did not earn much and were often unable to repay the loans. The students are victims here too, as we told them to do it.

Alas, none of the proposed student loan solutions involve fixing the underlying issue. If you said ‘we are sorry we pushed these loans on students and rewarded programs and institutions that do not deserve it, and we are going to stop giving loans for those programs and institutions and offer help to the suffering former students, ideally passing some of those costs on to the institutions’ then I would understand that. Instead, our programs are moving dollars mostly to relatively rich people who can afford to pay, and by offering forgiveness we are making the underlying problems far worse rather than better. Completely unacceptable even if it were constitutional.

Colorado governor Jared Polis, who really ought to know better, signs bipartisan bill to make first two years of college free for students whose family income is under $90k/year at in-state public schools. Technically this is 65 credits not counting AP/IB, concurrent enrollment, military credit or credit for prior learning, so there is even more incentive to get such credits.

The good news is they do have a full cliff, this falls off as you approach $90k, so they dodged the full version of quit-your-job insanity. The obvious bad news is that this is effectively one hell of a tax increase.

The less obvious bad news is this is setting up a huge disaster. Think about what the student who actually needs this help will do. They will go to a local college for two years for free. If they do well, they’ll get to 65 credits.

Then the state will say ‘oops, time to pay tuition.’ And what happens now? Quite a lot of them will choose to, or be forced to, leave college and get a job.

This is a disaster for everyone. The benefits of college mostly accrue to those who finish. At least roughly 25% of your wage premium is the pur...
New fast transformer inference ASIC — Sohu by Etched
by lemonhope — 2024-06-26T09:56:08.649Z
Link: https://www.etched.com/announcing-etched

I would bet that ASICs will run the roost in a few years and this is only the beginning.

They claim 500k tokens per second with Llama 70B.

Seems to be exactly what it looks like, an ASIC. Curious if this is somehow not what it looks like.
Empirical vs. Mathematical Joints of Nature
by Elizabeth — 2024-06-26T01:55:22.858Z

We (Alex and Elizabeth) are thinking about doing this project where we figure out how paradigm formation happened in chaos theory. Alex has also been thinking about paradigm formation for agent foundations (which people often talk about as being pre-paradigmatic). These are some thoughts on what paradigm formation means.

Alex_Altair

There's an issue which is something like, agent foundations is more like math than like science, and so it's unclear to me exactly what it means to have a paradigm. And I think this is similar to chaos theory, so it might be useful to talk it through and compare them.

Alex_Altair

On one hand, you could say that a paradigm is just a set of methods that people generally agree successfully solve problems. And in that sense, domains of math could totally have paradigms.

But in the sciences, there's another standard of whether a problem was solved, which is that reality was successfully predicted. Domains of math don't exactly try to "predict" phenomena, and so whether some problems were solved feels more subjective.

Alex_Altair

And I feel like both chaos theory and agent foundations are more like fields where people will just generally agree whether or not their confusion was resolved by certain frameworks, rather than fields where empirical phenomena are predicted.

Elizabeth

One definition I have been using of a paradigm is that it is a single solution to problems people believed were disparate. Newtonian physics predicted the rate of a falling apple and the orbit of the moon. Plate tectonics explained magnetic striping on the ocean floor, mountains, and the Wallace line (and let you use information about one to make predictions about the other that were born out).

Alex_Altair

Ah, interesting. That sounds related to something I've been thinking about which I might call paradigm shifts vs paradigm formation. I think Kuhn mostly talks about shifts, where there exists a previous strong paradigm. But if a field has no paradigm at all, then the formation of the first one might look different from a shift. Showing that multiple problems have a single solution sounds more like what paradigm formation might look like.

Elizabeth

I'm excited about chaos theory because it seems like it might be doing the same thing as the theory of plate tectonics- combining weather systems, eye movement, and a dripping faucet under a single set of equations, a

...
My Current Claims and Cruxes on LLM Forecasting & Epistemics
by ozziegooen — 2024-06-26T00:40:58.342Z

I think that recent improvements in LLMs have brought us to the point where LLM epistemic systems are starting to be useful. After spending some time thinking about it, I've realized that such systems, broadly, seem very promising to me as an effective altruist intervention area. However, I think that our community has yet to do a solid job outlining what this area could look like or figuring out key uncertainties.

This document presents a rough brainstorm on these topics. While I could dedicate months to refining these ideas, I've chosen to share these preliminary notes now to spark discussion. If you find the style too terse, feel free to use an LLM to rewrite it in a format you prefer.

I believe my vision for this area is more ambitious and far-reaching (i.e. not narrow to a certain kind of forecasting) than what I've observed in other discussions. I'm particularly excited about AI-heavy epistemic improvements, which I believe have greater potential than traditional forecasting innovations.

I'm trying to figure out what to make of this regarding our future plans at QURI, and I recommend that other organizations in the space consider similar updates.

Imaginary sketch of AI epistemic infrastructure, Dall-E 3

Key Definitions:

Epistemic process: A set of procedures to do analysis work, often about topics with a lot of uncertainty. This could be “have one journalist do everything themselves”, to a complex (but repeatable) ecosystem of multiple humans and software systems.
LLM-based Epistemic Process (LEP): A system that relies on LLMs to carry out most or all of an epistemic process. This might start at ~10% LLM-labor, but can gradually ramp up. I imagine that such a process is likely to feature some kinds of estimates or forecasts.
Scaffolding: Software used around an LLM system, often to make it valuable for specific use-cases. In the case of an LEP, a lot of scaffolding might be needed.

1. High-Level Benefits & Uses

Claim 1: If humans could forecast much better, these humans should make few foreseeable mistakes. This covers many mistakes, particularly ones we might be worried about now.

Someone deciding about talking to a chatbot that can be predicted to be net-negative (perhaps it would create an unhealthy relationship) could see this forecast and simply decide not to start the chat.
Say that a person’s epistemic state could follow one of four trajectories,

...
In favour of exploring nagging doubts about x-risk
by owencb — 2024-06-25T23:52:01.322Z

I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.

I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I’d encourage them to spend time trying to name their fears, and see what they think then. And I’d encourage them to talk about these things with other people, or write about the complexities of their thinking.

Partly this is because I don't expect people who are using intellectual arguments to override their gut to do a good job of consistently tracking what the most important things to do are on a micro-scale. So it would be good to get the different parts of them to sync more.

And partly because it seems like it would be a public good to explore and write about these things. Either their gut is onto something with parts of its scepticism, in which case it would be great to have that articulated; or their gut is wrong, but if other people have similar gut reactions then playing out that internal dialogue in public could be pretty helpful.

It's a bit funny to make this point about x-risk in particular because of course the above all applies to whatever topic. But I think people normally grasp it intuitively, and somehow that's less universal around x-risk. I guess maybe this is because people don't have any first-hand experience with x-risk, so their introductions to it are all via explicit arguments … and it's true that it's a domain where we should be unusually unwilling to trust our gut takes without hearing the arguments, but it seems to me like people are unusually likely to forget that they can know anything which has bearing on the questions without already being explicit (and also that perhaps the social environment, in encouraging people to take explicit arguments seriously, can accidentally overstep and end up discouraging people from taking a...
What is a Tool?
by johnswentworth — 2024-06-25T23:40:07.483Z

Throughout this post, we’re going to follow the Cognition -> Convergence -> Corroboration methodology[1]. That means we’ll tackle tool-ness in three main stages, each building on the previous:

Cognition: What does it mean, cognitively, to view or model something as a tool?
Convergence: Insofar as different minds (e.g. different humans) tend to convergently model the same things as tools, what are the “real patterns” in the environment which give rise to that convergence?
Corroboration: Having characterized the real patterns convergently recognized as tool-ness, what other properties or implications of tool-ness can we derive and check? What further predictions does our characterization make?

We’re not going to do any math in this post, though we will gesture at the spots where proofs or quantitative checks would ideally slot in.

Cognition: What does it mean, cognitively, to view or model something as a tool?

Let’s start with a mental model of (the cognition of) problem solving, then we’ll see how “tools” naturally fit into that mental model.

When problem-solving, humans often come up with partial plans - i.e. plans which have “gaps” in them, which the human hasn’t thought through how to solve, but expects to be tractable. For instance, if I’m planning a roadtrip from San Francisco to Las Vegas, a partial plan might look like “I’ll take I-5 down the central valley, split off around Bakersfield through the Mojave, then get on the highway between LA and Vegas”. That plan has a bunch of gaps in it: I’m not sure exactly what route I’ll take out of San Francisco onto I-5 (including whether to go across or around the Bay), I don’t know which specific exits to take in Bakersfield, I don’t know where I’ll stop for gas, I haven’t decided whether I’ll stop at the town museum in Boron, I might try to get pictures of the airplane storage or the solar thermal power plant, etc. But I expect those to be tractable problems which I can solve later, so it’s totally fine for my plan to have such gaps in it.

Bakersfield is roughly where the “9 hr 42 min” box is.

How do tools fit into that sort of problem-solving cognition?

Well, sometimes similar gaps show up in many different plans (or many times in one plan). And if those gaps are similar enough, then it might be possible to solve them all “in the same way”. Sometimes we can even build a physical object which makes it easy to...
When do alignment researchers retire?
by Jordan Taylor — 2024-06-25T23:30:25.520Z

At what point will it no longer be useful for humans to be involved in the process of alignment research? After the first slightly-superhuman AGI, well into superintelligence, or somewhere in between?

Feel free to answer differently for different kinds of human involvement:

Humans could be involved as a source of data about human values
Humans could be involved as a red-team, trying to get evidence of misalignment or to verify the trustworthiness of systems
Humans could be involved in setting the broad research agenda, delegating to the AGIs
Humans could be involved in complementing the technical weaknesses of the AGIs, helping them in some way to research new alignment methods

What do you envision we are doing between AGI and superintelligence?
Compute Governance Literature Review 
by sijarvis — 2024-06-25T22:41:46.694Z

This white paper reviews the use of compute governance to mitigate risks from AI, the reasoning behind this approach, the ways in which it has been used and in which it could be further developed. It aims to provide an introduction for individuals who feel that there is a need for AI safety and would like to further understand this governance approach. It was prepared as a final project for the BlueDot Impact AI Alignment Course.

Executive Summary

Compute governance is a method for monitoring and regulating AI systems by tracking the hardware required to develop frontier models. At its most simple it can be implemented by recording the owners and locations of physical chips and it is also possible to be extended to require functionality that enables reporting or other security features. It can assist the enforcement of existing regulations that require companies to report AI system development over certain thresholds. There are many reasons why compute governance is seen as a promising approach, however it does not mitigate all possible risks from AI and is likely to be most effective as part of a package of governance interventions.

Figure 1 - AI Generated Image of a chip secured with a lock, Copilot Designer

What is Compute?

The training process for an AI model involves calculating the result of many mathematical equations. Depending on the context, compute can be used to mean either, the total number of calculations used to train a model (measured in floating-point operations or FLOP), how many such calculations are performed over time (in FLOPS, floating-point operations per second), or more generally the resources and hardware used to perform the calculations.

Large Language Models (LLMs) and multimodal models require immense computational power to train on massive source datasets, to access the level of compute required powerful Graphics Processing Units (GPUs), Tensor Processing Units (TPUs) or other specialised AI hardware chips are used. It is estimated that GPT-4 required 2.1x1025 FLOP for training (Sevilla et al., 2022) whereas cutting-edge AI chips have performance levels measured in teraFLOPS or 1012 FLOPS (H100 Tensor Core GPU | NVIDIA, n.d.) meaning that tens of thousands of chips are required for each training run to be completed in a reasonable amount of time.

The outcome of this is that compute is intrinsically linked to the access...
Computational Complexity as an Intuition Pump for LLM Generality


by aribrill — 2024-06-25T20:25:36.751Z

With sufficient scale and scaffolding, LLMs will improve without bound on all tasks to become superhuman AGI, to the extent they haven’t already. No, wait! LLMs are dead-end pattern-matching machines fundamentally incapable of general reasoning and novel problem solving. Which is it?

I’ll call these opposing points of view “LLM Scaler” and “LLM Skeptic”. The first appears to be held by the big AGI labs and was recently exemplified by Leopold Aschenbrenner’s

Situational Awareness

series, while the second is influenced by cognitive science and can be associated with researchers such as François Chollet, Gary Marcus, and Melanie Mitchell. This post loosely generalizes these two stances, so any misrepresentation or conflation of individuals’ viewpoints is my own. I recommend Egg Syntax’s

recent post

for further introduction and a summary of relevant research. We can caricature the debate something like this:

LLM Scaler: LLMs will take us to AGI. See straight lines on graphs. See correct answers to hard questions on arbitrary topics. See real customers paying real money for real value.

LLM Skeptic: LLMs achieve high skill by memorizing patterns inherent in their giant training set. This is not the path to general intelligence. For example, LLMs will never solve Task X.

LLM Scaler: They solved it.

LLM Skeptic: No kidding? That doesn’t count then, but they’ll really never solve Task Y. And you can’t just keep scaling. That will cost trillions of dollars.

LLM Scaler: Do you want to see my pitch deck?

Meanwhile, on Alternate Earth, Silicon Valley is abuzz over recent progress by Large List Manipulators (LLMs), which sort a list by iteratively

inserting

each item into its correct location. Startups scramble to secure special-purpose hardware for speeding up their LLMs.

LLM Scaler: LLMs are general list sorters, and will scale to sort lists of any size. Sure, we don’t quite understand how they work, but our empirical compute-optimal scaling law (N ~ C^0.5) has already held across a dozen OOMs, and we’re spending billions in venture capital to keep it going!

LLM Skeptic: That absurd expense is unsustainable. Can’t you see that? There’s no way that LLMs are truly general list sorters. Good luck getting one to sort a list with a million items.

LLM Scaler: We already have.

LLM Skeptic: Oh. Well then, LLMs will never sort a list with a BILLION items!

The “LLM” Skep...
Failure Modes of Teaching AI Safety
by Eleni Angelou — 2024-06-25T19:07:46.826Z

Why I'm writing this

I'm about to teach my AI safety course for the fourth time. As I'm now updating the syllabus for the upcoming semester, I summarize my observations on what can go wrong when teaching AI safety. These have mostly not happened during my teaching but are generally likely to happen - as more AIS courses are developed and taught around the world - and I've especially thought about them when preparing the course so that they don't happen.

1. Alignment feels like a lost cause

Depending on how x-risk is presented, getting misaligned AI might appear as an inevitable future since the problem is too complex and hard, and there isn't enough time for AI alignment research to generate robust techniques.

How to avoid: make sure to emphasize all the work on alignment and governance and how your students could also be doing some of that (if they wanted to). I'm also updating my syllabus to include the discussion about AI Pause.

2. There is no (historical/philosophical) context

Just like with all complex ideas, situating the problem within its context can make a big difference in how it will be perceived. Talking about AI systems that all of a sudden become a threat to humanity is confusing, to say the least.

How to avoid: talk about the foundations of AI, the debate between symbolic AI and connectionism in cognitive science, and set the stage for how we got to contemporary LLMs.

3. Your audience gets (disproportionately more) excited about capabilities

This is more common with people with technical backgrounds who like to build tools and applications and are blindly or naively excited about technological progress.

How to avoid: clearly explain the conceptual parts of the problem, describe it in more industry-friendly terminology, provide a lot of examples.

4. But ... what about sexist/racist etc. algorithms?

Hinting at AI ethics being more important than this alignment story. Also, appears in the context of the Longtermism debate.

How to avoid: show how this is a form of misalignment, happening right now. AI ethics should not be opposing AI safety.

5. There are no actionable next steps

This is true for many courses. Once you've finished it it's just another letter on your transcript. I think that there are good reasons for an AIS course in particular not to fall into that category.

How to avoid: connect the mat...
Kingfisher Summer Tour 2024
by jefftk — 2024-06-25T18:50:02.253Z

Kingfisher is going
on tour again:

Thr
2024-06-27

Rochester NY

Fri
2024-06-28

Pittsburgh PA

Sat
2024-06-29

Bloomington IN

Sun
2024-06-30

St Louis MO

Mon
2024-07-01

Cincinnati OH

Tue
2024-07-02

Indianapolis IN

Cecilia made a flyer:

Harris will be calling,
and we'll be driving together. Anna and Lily are coming too, which
I'm excited about.

It's about 40hr of driving, which is a lot, but we're spreading it
over eight days and we do have some relatively light days, so it
should be ok:

If you're somewhere along here, I'd love to see you; come dance!
Incentive Learning vs Dead Sea Salt Experiment
by Steven Byrnes — 2024-06-25T17:49:01.488Z

(Target audience: People both inside and outside neuroscience & psychology. I tried to avoid jargon.)

1. Background context

One of my interests is the neuroscience of motivation. This is a topic I care about for the same obvious reason as everyone else: to gain insight into how to control the motivations of future superintelligent brain-like Artificial General Intelligence.

(Wait, you’re telling me that there are other reasons people care about the neuroscience of motivation? Huh. No accounting for taste, I guess.)

Part of the neuropsychology-of-motivation literature is a few related phenomena variously called “incentive learning”, “retasting”, “devaluation”, and “revaluation”. In the experiments of interest here:

You change something about an animal’s homeostatic state (e.g. its level of hunger),
…And you see whether the animal accounts for that change in its behavior (e.g. whether it tries to get food).

For example, my own children, despite many years of experience, are very shaky on this learning task. After skipping a meal, they get grumpy rather than hungry, and when they finally sit down and take a bite, they’re shocked, shocked, to find how much they were craving it.

Anyway, there is a massive literature on these phenomena. You won’t learn any practical parenting tips, alas, but you can learn lots of random things like ‘what happens to revaluation if we block the activity of neurotransmitter X in macaque brain region Y?’. And there are intriguing contrasts, where the animal does or doesn’t adapt its behavior to its physiological state in two situations that seem generally pretty similar. The title of this post is an example—I’ll go over it in Section 2.

People do these studies out of the (reasonable) expectation that they will help illuminate the nuts-and-bolts of how homeostatic needs connect to actions, decisions, and desires. But unfortunately, there’s a generic problem: it’s hard to go backwards from observed behaviors to how brain algorithms generated those behaviors. You basically need to already have a good theoretical framework. My spicy opinion is that most people in the field don’t, and that they fill in that gap by grasping for salient but unhelpful concepts.

In some cases, these unhelpful concepts are borrowed from AI—e.g., “model-based RL versus model-free RL” is a salient partitioning for AI practitioners, whether...