trentmkelly/LessWrong-43k huggingface dataset:
Making every researcher seek grants is a broken model
by jasoncrawford — 2024-01-26T16:06:26.688Z
Link: https://rootsofprogress.org/the-block-funding-model-for-science

When Galileo wanted to study the heavens through his telescope, he got money from those legendary patrons of the Renaissance, the Medici. To win their favor, when he discovered the moons of Jupiter, he named them the Medicean Stars. Other scientists and inventors offered flashy gifts, such as Cornelis Drebbel’s perpetuum mobile (a sort of astronomical clock) given to King James, who made Drebbel court engineer in return. The other way to do research in those days was to be independently wealthy: the Victorian model of the gentleman scientist.

Galileo demonstrating law of gravity in presence of Giovanni de' Medici, 1839 fresco by Giuseppe Bezzuoli Meisterdrucke

Eventually we decided that requiring researchers to seek wealthy patrons or have independent means was not the best way to do science. Today, researchers, in their role as “principal investigators” (PIs), apply to science funders for grants. In the US, the NIH spends nearly $48B annually, and the NSF over $11B, mainly to give such grants. Compared to the Renaissance, it is a rational, objective, democratic system.

However, I have come to believe that this principal investigator model is deeply broken and needs to be replaced.

That was the thought at the top of my mind coming out of a working group on “Accelerating Science” hosted by the Santa Fe Institute a few months ago. (The thoughts in this essay were inspired by many of the participants, but I take responsibility for any opinions expressed here. My thinking on this was also influenced by a talk given by James Phillips at a previous metascience conference. My own talk at the workshop was written up here earlier.)

What should we do instead of the PI model? Funding should go in a single block to a relatively large research organization of, say, hundreds of scientists. This is how some of the most effective, transformative labs in the world have been organized, from Bell Labs to the MRC Laboratory of Molecular Biology. It has been referred to as the “block funding” model.

Here’s why I think this model works:

Specialization

A principal investigator has to play multiple roles. They have to do science (researcher), recruit and manage grad students or research assistants (manager), maintain a lab budget (administrator), and write grants (fundraiser). These are different roles, and not everyone has the skill or inclination to do them all. The university model adds...
Notes on Innocence
by David Gross — 2024-01-26T14:45:45.612Z

This post examines the virtue of innocence. As with my other posts in this sequence, I’m less interested in breaking new ground and more in synthesizing any wisdom I could find on the subject. I wrote this not as an expert, but as someone who wants to learn more. I hope it helps people who want to know more about this virtue and how to nurture it.

I felt more ambivalence about this virtue than with most, so you may find this post unsatisfactorily waffley.

What is this virtue?

The virtue of innocence I want to investigate is not the legal “innocent until proven guilty” kind, but “the sweet converse of an innocent mind”[1] kind: more of a character trait than a verdict. I mean innocence not in contrast to guilt, but in contrast to guile.

A lot of what has been written about a virtue called “innocence” uses the word as a euphemism for virginity or for sexual obliviousness, often specifically for these things in girls and women. I want to consider instead a more-rarely discussed virtue of innocence—a character trait that is also available to sexually-mature adults of all genders.

I’ll first introduce a couple of scenarios to illustrate the sort of innocence I have in mind.

Scenario #1: The racist joke

Alice tells a racist joke to her acquaintances, Carol & Bob: “What do you call a black abortion clinic? Crime Stoppers.” While Carol is considering how to most diplomatically and unambiguously show her disapproval, Bob abruptly says: “I don’t get it.”

Alice: “It’s a joke.”

Bob: “Yeah, I know. But I don’t get it. What does the color of the clinic have to do with calling it ‘Crime Stoppers’?”

Alice: “No: the abortion clinic isn’t black; it’s an abortion clinic for black people.”

Bob: “Do they even have those?”

Alice: “It’s a joke. You’re just supposed to pretend. Imagine if.”

Bob: “I still don’t get it.”

Alice: “Well you’ve ruined it now. If I have to explain it, it won’t be funny.”

The “joke” Alice told—even if you put aside the offensiveness of it—isn’t funny. It isn’t really a joke, but a shibboleth in the form of a joke. What Alice is communicating is something like this: “We all understand that black people are a bunch of criminals, don’t we?” What Carol is trying to come up with is something like: “I don’t approve of that stereotype or think it’s appropriate to promulgate it.” Bob simply says “nope” ¯\_(ツ)_/¯.

There’s something attractive about Bob’s innocence. While ...
Stacked Laptop Monitor
by jefftk — 2024-01-26T14:10:05.297Z

A few weeks ago I

broke my
laptop's screen and was speculating on whether

putting a portable monitor
above the screen would do what I wanted. The hope was that it
would let me set up a laptop where both my wrists and neck were in
comfortable positions. It does!

The monitor is a 1080p
model that gets it's power and video signal over USB-C. It's
screwed to an aluminum plate with M4 VESA screws, which is connected
to another plate with butler tray hinges:

Then the bottom plate velcros to the back of the laptop screen:

I probably should have gotten some kind of hinge that locks, but since
I didn't I cut a piece of wood to chock it:

I should sort out something sturdier and harder to lose.

It folds up, with the portable monitor's cover protecting the screen:

It still fits in my backpack's laptop pocket, but only just.

I tried it on the airplane yesterday, and it's so much better
ergonomically!

It also doesn't need the chock, since to fold up the seat in front of
me would need to not be there.

There are a few downsides, though:

I need to be careful to position it so all the forces are
aligned, and only use it in places where it's not going to get
jostled. Building a kickstand (which all the others seem to have)
would probably help a lot with this.

If you use it in a meeting you can't see the person across from
you:

If you're nine it's probably too high for you:

I've only used it a bit, but so far I'm very happy with it. I'm
composing this post on it while traveling, and it's way more
comfortable.

Comment via: facebook, mastodon
Surgery Works Well Without The FDA
by Maxwell Tabarrok — 2024-01-26T13:31:29.968Z
Link: https://maximumprogress.substack.com/p/surgery-works-well-without-the-fda

Here is a conversation from the comments of

my last post on the FDA

with fellow

progress blogger Alex Telford

that follows a pattern common to many of my conversations about the FDA:

Alex: Most drugs that go into clinical trials (90%) are less effective or safe than existing options. If you release everything onto the market you'll get many times more drugs that are net toxic (biologically or financially) than the good drugs you'd get faster. You will almost surely do net harm.

Max: Companies don't want to release products that are worse than their competitors.

Companies test lots of cars or computers or ovens which are less effective or safe than existing options but they only release the ones that are competitive. This isn't because most consumers could tell whether their car was less efficient or that their computer is less secure, and it's not because making a less efficient car or less secure computer is against the law.

Pharmaceutical companies won't go and release hundreds of dud or dangerous drugs just because they can. That would ruin their brand and shut down their business. They have to sell products that people want.

Alex: Consumer products like ovens and cars aren't comparable to drugs. The former are engineered products that can be tested according to defined performance and safety standards before they are sold to the public. The characteristics of drugs are more discovered than engineered. You can't determine their performance characteristics in a lab, they can only be determined through human testing (currently).

Alex claims that without the FDA, pharmaceutical companies would release lots of bunk drugs. I respond that we don’t see this behavior in other markets. Car companies or computer manufacturers could release cheaply made, low quality products for high prices and consumers might have a tough time noticing the difference for a while. But they don’t do this, they always try to release high quality products at competitive prices.

Alex responds, fairly, that car or computer markets aren’t comparable to drug markets. Pharmaceuticals have stickier information problems. They are difficult for consumers to evaluate and, as Alex points out, usually require human testing.

This is usually where the conversation ends. I think that consumer product markets are informative for what free-market pharmaceuticals would look like, Alex (and lots of other reason...
Workshop (hackathon, residence program, etc.) about for-profit AI Safety projects?
by Roman Leventov — 2024-01-26T09:49:30.836Z

Reading posts like "This might be the last AI Safety Camp" makes me feel very sad. The degree to which AI Safety is currently funding-constrained is clearly inadequate on the national and civilisational levels.

Maybe let's try to make a smart counter-move and accelerate the development of for-profit AI Safety projects (see also the comments to that post, and this post)? With the obvious idea to pull some VC money, which is a different pool than AI safety philanthropic funds.

Potential collaborations:

https://vc.ae.studio/
https://www.lionheart.vc/
https://www.mythos.vc/
https://www.joinef.com/ (or maybe they are working on something like this already? At least Matt Clifford was the UK PM's sherpa for AI Safety Summit, and now working on AI full time)
More?

P.S. I'm not a professional organiser or community builder, nor a startup accelerator program manager, so just floating the idea, but I'd be very eager to participate if something like this is organised.
Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI
by Jeremy Gillen — 2024-01-26T07:22:06.370Z

A pdf version of this report is available here.

Summary

In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.

In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.

In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order to achieve an outcome. This is because the task has to be specified in terms of outcomes, and the AI needs to be robust to new obstacles in order to achieve these outcomes.

In Section 3 we argue that novel science will necessarily require the AI to learn new things, both facts and skills. This means that an AI’s capabilities will change over time which is a source of dangerous distribution shifts.

In Section 4 we further argue that training methods based on external behavior, which is how AI systems are currently created, are an extremely imprecise way to specify the goals we want an AI to ultimately pursue. This is because there are many degrees of freedom in goal specification that aren’t pinned down by behavior. AIs created this way will, by default, pursue unintended goals.

In Section 5 we discuss why we expect oversight and control of powerful AIs to be difficult. It will be difficult to safely get useful work out of misaligned AIs while ensuring they don’t take unwanted actions, and therefore we don’t expect AI-assisted research to be both safe and much faster than current research.

Finally, in Section 6 we discuss the consequences of building a powerful AI with improperly specified goals. Such an AI could likely escape containment measures given realistic levels of security, and then pursue outcomes in the world that would be catastrophic for humans. It seems very unlikely that these outcomes would be compatible with human empowerment or survival.

Introduction

We expect future AI systems will be a...
Approximately Bayesian Reasoning: Knightian Uncertainty, Goodhart, and the Look-Elsewhere Effect
by RogerDearnaley — 2024-01-26T03:58:16.573Z

Approximate Bayesian Reasoning

AIXI is well-known model of a Bayesian reasoner. However, it's not a practical one, since both the Solomonoff prior's basis, Kolmogorov complexity, and the rest of the Solomonoff induction process of Bayesian reasoning as used by AIXI are uncomputable: they require unlimited computational resources. [Even the carefully more limited case of attempting to approximate using polynomial computational resources the polynomial-limited-Kolmogorov complexity of a program for outputting the object in polynomial time turns out to raise some fascinating issues in computational complexity theory.] Any real-world agent will need to figure out how to make do with finite computational resources, just as humans practitioners of STEM and rationalism do. This introduces a wide range of concerns into our approximate Bayesianism:

Missing hypotheses: it is quite likely that the true hypothesis is not even in the set of hypotheses that we are currently considering. About the best we can hope for is that one or more of the hypotheses we're considering is a moderately good fit for reality over some range of world states, including ones in and perhaps close to the distribution of states that we've already encountered and used as evidence.
Inaccurate priors: the Solomonoff prior is uncomputable, cannot even be polynomially approximated reliably, and well-know arguments such as uniform priors, maximum entropy priors, and symmetry-group based measure priors are only accurately usable if you actually have the full hypothesis space available. There are ways to make better and worse guesses of priors, but we should reasonably expect that our prior was wrong, too high or too low (more likely too high because we were unaware of part of the hypothesis space). In practice, as we accumulate evidence, this problem gradually gets overcome by the weight of evidence, but in the meantime our posteriors will be wrong too. So a hypothesis in our hypothesis set basically represents no only itself, but also other hypotheses we have not yet considered that are similar enough to it (at least in some region) to not be significantly distinguishable under any evidence we yet have, and as and when we come up with such hypotheses, splitting its prior/posterior between them in some principled-looking way might be a reasonable choice of new prior. So for example, when adding Einstein'

...
RAND report finds no effect of current LLMs on viability of bioterrorism attacks
by StellaAthena — 2024-01-25T19:17:30.493Z
Link: https://www.rand.org/pubs/research_reports/RRA2977-2.html

Key Findings

This research involving multiple LLMs indicates that biological weapon attack planning currently lies beyond the capability frontier of LLMs as assistive tools. The authors found no statistically significant difference in the viability of plans generated with or without LLM assistance.
This research did not measure the distance between the existing LLM capability frontier and the knowledge needed for biological weapon attack planning. Given the rapid evolution of AI, it is prudent to monitor future developments in LLM technology and the potential risks associated with its application to biological weapon attack planning.
Although the authors identified what they term unfortunate outputs from LLMs (in the form of problematic responses to prompts), these outputs generally mirror information readily available on the internet, suggesting that LLMs do not substantially increase the risks associated with biological weapon attack planning.
To enhance possible future research, the authors would aim to increase the sensitivity of these tests by expanding the number of LLMs tested, involving more researchers, and removing unhelpful sources of variability in the testing process. Those efforts will help ensure a more accurate assessment of potential risks and offer a proactive way to manage the evolving measure-countermeasure dynamic.

The linkpost is to the actual report, see also their press release.
Bayesian Reflection Principles and Ignorance of the Future
by  — 2024-01-25T19:00:16.463Z

Hi everyone,

New user here. I am learning Bayesian epistemology by working through Mike Titelbaum’s recent Fundamentals of Bayesian Epistemology. I’ve been learning about deference principles — principles concerning deferring to experts, where experts may include experts in the colloquial sense, the objective chances, and even one’s future self (provided one is sure that one’s opinions will update via conditionalization).

I have a question about the Reflection Principle. Roughly, Reflection tells us to adopt the credence now that we expect we would adopt at some future time (on the assumption that our opinions are updated by conditionalizing). More carefully, Reflection says:

Reflection: For any proposition A in L, real number x, and times ti and tj with j > i, rationality requires: cri(A | crj(A) = x) = x (Titelbaum, 143).

Obviously, I do not have perfect knowledge of what my future evidence will be, however, so I cannot be sure what my future credences will be, de re. (Of course, I could know them de dicto, as in: “my future credences will be whatever credences result from conditionalizing on the evidence I acquire.”)

Suppose that currently my evidence is E and cr2024(p) = 0.5. In 2050, I might have evidence E (i.e. my evidence won't have changed), E*, or E**. If In 2050, I have evidence E, then cr2050(p) = 0.5. But suppose that I instead 2050, I have evidence E*, and by conditionalizing on E* I have cr2050(p) = 0.3. Or suppose that in 2050, I have evidence E**, and by conditionalizing on E** I have cr2050(p) = 0.7. (Suppose these are the only three possibilities, just for the sake of ease.)

I don't in 2024 know what evidence I'll have as to whether p in 2050. To what future self should my 2024 self defer, then? The one with E, E*, or E**? Is there a way to adjudicate this? The intuitive answer is that I should figure out which is likeliest: (i) that my future self has E, (ii) that my future self has E*, or (iii) that my future self has E**. I can then defer to the likeliest future self. But how can I determine this?

I apologize if this is a n00b question. Titelbaum does not discuss this issue in the book, and I'm not sure where to look.

Thanks.
"Does your paradigm beget new, good, paradigms?"
by Raemon — 2024-01-25T18:23:15.497Z

A very short version of this post, which seemed worth rattling of quickly for now.

A few months ago, I was talking to John about paradimicity in AI alignment. John says "we don't currently have a good paradigm." I asked "Is 'Natural Abstraction' a good paradigm?". He said "No, but I think it's something that's likely to output a paradigm that's closer to the right paradigm for AI Alignment."

"How many paradigms are we away from the right paradigm?"

"Like, I dunno, maybe 3?" said he.

Awhile later I saw John arguing on LessWrong with (I think?) Ryan Greenblatt about whether Ryan's current pseudo-paradigm was good. (Sorry if I got the names here or substance here wrong, I couldn't find the original thread, and it seemed slightly better to be specific so we could dig into a concrete example).

One distinction in the discussion seemed to be something like:

On one hand, Ryan thought his current paradigm (this might have been "AI Control", as contrasted with "AI Alignment") had a bunch of traction on producing a plan that would at least reasonably help if we had to align superintelligent AIs in the near future.
On the other hand, John argued that the paradigm didn't feel like the sort of thing that was likely to bear the fruit of new, better paradigms. It focused on an area of the superintelligence problem that, while locally tractable, John thought was insufficient to actually solve the problem, and also wasn't the sort of thing likely to pave the way to new paradigms.

Now a) again I'm not sure I'm remembering this conversation right, b) whether either of those points are true in this particular case would be up for debate and I'm not arguing they're true. (also, regardless, I am interested in the idea of AI Control and think that getting AI companies to actually do the steps necessary to control at least nearterm AIs is something worth putting effort into)

But it seemed good to promote to attention the idea that: when you're looking at clusters of AI Safety research and thinking about whether it is congealing into a useful, promising paradigm, one of the questions to ask is not just "does this paradigm seem locally tractable" but "do I have a sense that this paradigm will open up new lines of research that can lead to be better paradigms?".

(Whether one can be accurate in answering that question is yet another uncertainty. But, I think if you ask yourself "is this app...
AI #48: The Talk of Davos
by Zvi — 2024-01-25T16:20:26.625Z

While I was in San Francisco, the big head honchos headed for Davos, where AI was the talk of the town. As well it should be, given what will be coming soon. It did not seem like anyone involved much noticed or cared about the existential concerns. That is consistent with the spirit of Davos, which has been not noticing or caring about things that don’t directly impact your business or vibe since (checks notes by which I mean an LLM) 1971. It is what it is.

Otherwise we got a relatively quiet week. For once the scheduling worked out and I avoided the Matt Levine curse. I’m happy for the lull to continue so I can pay down more debt and focus on long term projects and oh yeah also keep us all farther away from potential imminent death.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Might not come cheap.

Language Models Don’t Offer Mundane Utility. The ancient art of walking.

Copyright Confrontation. It knows things, but it still cannot drink.

Fun With Image Generation. Poisoning portraits in the park.

Deepfaketown and Botpocalypse Soon. Use if and only if lonely.

They Took Our Jobs. The one saying it won’t happen interrupted by one doing it.

Get Involved. New jobs, potential unconference.

In Other AI News. Various people are doing it, for various values of it.

Quiet Speculations. How fast is efficiency improving?

Intelligence Squared. Why so much denial that importantly smarter is possible?

The Quest for Sane Regulation. New polls, new bad bills, EU AI Act full text.

Open Model Weights Are Unsafe and Nothing Can Fix This. More chips, then.

The Week in Audio. Nadella, Altman and more.

Rhetorical Innovation. Are you for or against the existence of humanity?

Malaria Accelerationism. All technology is good, you see, well, except this one.

Aligning a Smarter Than Human Intelligence is Difficult. Diversification needed.

Other People Are Not As Worried About AI Killing Everyone. Anton and Tyler.

The Lighter Side. No spoilers.

Language Models Offer Mundane Utility

Say you can help people respond to texts on dating apps via a wrapper, charge them $28/month, claim you are making millions, then try to sell the business for $3.5 million. Why so much? Classic black market situation. The readily available services won’t make it easy on you, no one reputable w...