trentmkelly/LessWrong-43k huggingface dataset:
On the subject of in-house large language models versus implementing frontier models
by Annapurna — 2024-09-23T15:00:32.811Z

A recent survey in the US states that 39.4% of adults are using generative AI for tasks both at work and outside of work, which highlights the rapidly increasing dependency on these models. Anecdotally, I introduced my wife to chatGPT four months ago, and today she consistently consults chatGPT 4o not only for work related matters, but also for day-to-day tasks at home. In our household, googling is slowly becoming a thing of the past.

Over the last year and a half, I have heard lots of chatter of regular businesses hiring teams of engineers to design in-house LLM applications. The arguments for building in-house LLMs are obvious: You control the architecture, data, and sensitive information of your business, as opposed to exposing this data to these ‘black box’ models. A year ago, it seemed like a good tradeoff to make, but after GPT-4 and other frontier models have been released, it seems to me that any regular business that continues to develop LLMs in-house will be left behind. Frontier models have advanced so quickly in terms of complexity, data scale, and efficiency that matching this pace internally may no longer be feasible for most regular businesses.

Looking into the future, I am curious about the following:
Aside from the reasons stated above, are there any other reasons why regular businesses should be spending resources creating their own in-house LLMs?
Is there a way to identify which companies are partnering with the builders of frontier models versus those developing their own in-house models within an industry?

If one of the barriers of customizing and implementing a frontier LLM within a firm is data cleaning, as sarahconstatin mentions in this post, is there a business opportunity in becoming a data cleaner, aka the bridge between regular companies and frontier model builders?
A basic systems architecture for AI agents that do autonomous research
by Buck — 2024-09-23T13:58:27.185Z

A lot of threat models describing how AIs might escape our control (e.g. self-exfiltration, hacking the datacenter) start out with AIs that are acting as agents working autonomously on research tasks (especially AI R&D) in a datacenter controlled by the AI company. So I think it’s important to have a clear picture of how this kind of AI agent could work, and how it might be secured. I often talk to people who seem to have a somewhat confused picture of how this kind of agent setup would work that causes them to conflate some different versions of the threat model, and to miss some important points about which aspects of the system are easy or hard to defend.

So in this post, I’ll present a simple system architecture for a system where autonomous AIs are doing AI R&D research, and then draw diagrams describing how various threat scenarios involving misaligned AI involve compromises in different places.

The main point I want to make here is that the following responsibilities will almost surely be hosted on logically different computers, and probably on different physical machines:

The inference server that serves the LLM API
The scaffold server, which maintains the state of an agent loop; this is responsible for repeatedly calling the inference server, parsing the LLM response, and executing actions with the tools the agent has access to.
The execution server, on which the agent can run code and see its results.

I think that this is both:

Pretty likely to happen even if everyone involved is being pretty careless. Everything I’m talking about here is pretty standard. E.g. it’s how the o1 agent evaluations were run, how METR runs their evals, how I assume ChatGPT with Code Interpreter is architected, et cetera.
A good starting point if you want your agent to be at all secure or controlled.

People often talk about “the machine that the AI agent is running on”; I think it’s actually pretty crucial to specify which of these different machines you’re talking about. For example, when my Claude agent is running on my laptop, it’s not in a very good position to exfiltrate its weights, because even though the inference is happening inside Anthropic’s datacenter and has access to its weights, the agent is executing code on my laptop, which has no access to the Anthropic datacenter—it could theoretically try to hack its way in, but it has no advantages...
Book Review: On the Edge: The Fundamentals
by Zvi — 2024-09-23T13:40:11.058Z

The most likely person to write On the Edge was Nate Silver.

Grok thinks the next most likely was Michael Lewis, followed by a number of other writers of popular books regarding people thinking different.

I see why Grok would say that, but it is wrong.

The next most likely person was Zvi Mowshowitz.

I haven’t written a book for this type of audience, a kind of smarter business-book, but that seems eminently within my potential range.

On the Edge is a book about those living On The Edge, the collection of people who take risk and think probabilistically and about expected value. It centrally covers poker, sports betting, casinos, Silicon Valley, venture capital, Sam Bankman-Fried, effective altruism, AI and existential risk.

Collectively, Nate Silver calls this cultural orientation The River.

It is contrasted with The Village, which comprises roughly the mainstream mostly left-of-center institutions, individuals and groups that claim that they are The Experts and the Very Serious People.

If you are thinking about Secret Third Thing, that Village plus River very much does not equal America, I was thinking a lot about that too. Hold that thought.

The book is a collection of different topics. So this review is that, as well.

The central theme here will be Yes, And.

Nate Silver wrote On the Edge for people who are not in The River. I suspect his main target was The Village, but there are also all the people who are neither. He knows a lot more than he is saying here, but it is a popular book, and popular books have to start at the beginning. There was a lot to cover.

This joy of this review? I don’t have to do any of that. This is aimed at those who read things I write. Which means most of you already know a lot, often most, of what is in at least large portions of On the Edge.

So this is a chance to go deeper, be more detailed and opinionated, with a different world model in many ways, and expertise in different spots along the River.

As with my other book reviews, quotes by default are from the book, and the numbers in parenthesis are book locations on Kindle. I sometimes insert additional paragraph breaks, and I fix capitalization after truncating quotes while being careful to preserve original intent. Also notice that I change around the order when it improves flow.

This review is in four parts, which I plan to post throughout the week: The Fundamentals (this post),...
Switching to a 4GB SD
by jefftk — 2024-09-23T11:20:05.432Z

When I initially switched the computer portion of my

rhythm stage setup over to a Raspberry PI I
went with a 32GB SD card because

that's
recommended. But I don't need very much space for what I'm doing
and a bigger SD card makes copies slow: the cheap cards I've been
using take me

over an
hour to write with my Mac's card reader.

I recently made some changes to my setup, removing support
for bass whistle on the PI (now that I have an arduino-based version) and adding
support for more
foot-bass options. This normally means round of re-imaging all of
my SD cards: I have a bunch of them because SD card corruption is
probably the most common reason my system breaks [1], and it's easy to
have extra cards. I'm pretty sick of 82-minute waits, however, so I
decided to go down to four-GB SD cards.

I imaged a card with Raspberry
Pi OS Lite (64-bit), started setting it up, and immediately ran
out of space. Turns out apt-get update and apt-get
upgrade can use a lot of space. After apt-get
purge and apt-get clean, however, I had about
300MB to spare. It would probably be possible to get a lite
distribution to go a lot smaller than this, since the only actually
large thing I need is the sound font, fluid-soundfont-gm
at 145MB, but this seems to work fine.

Regardless, it works great, and I can now image a card in 15min:

$ time sudo dd if=~/Desktop/rpi-4gb.dmg of=/dev/disk4
7782400+0 records in
7782400+0 records out
3984588800 bytes transferred in 877.723067 secs (4539688 bytes/sec)

real 14m41.556s
user 0m3.299s
sys 0m28.741s

[1] This used to happen a lot, but now mostly doesn't. The hardware
hasn't changed, so I think the reason it has stopped happening is that
my Raspberry PI is no longer responsible for the bass whistle. My
MIDI router doesn't write to the file system, and doesn't read beyond
initialization, while the bass whistle version used the file system to
persist settings across reboots.

Comment via: facebook, mastodon
Model evals for dangerous capabilities
by Zach Stein-Perlman — 2024-09-23T11:00:00.866Z

Testing an LM system for dangerous capabilities is crucial for assessing its risks.

Summary of best practices

Best practices for labs evaluating LM systems for dangerous capabilities:

Publish results
Publish questions/tasks/methodology (unless that's dangerous, e.g. CBRN evals; if so, offer to share more information with other labs, government, and relevant auditors, and publish a small subset)
Do good elicitation and publish details (or at least demonstrate that your elicitation is good):

General finetuning (

for

"instruction following, tool use, and general agency" and maybe capabilities in the relevant area)
Helpful-only; no inference-time mitigations
Scaffolding, prompting, chain of thought

The lab should mention some details so that observers can understand how powerful and optimized the scaffolding is. Open-sourcing scaffolding or sharing techniques is supererogatory. If the lab does not share its scaffolding, it should show that the scaffolding is effective by running the same model with the most powerful relevant open-source scaffolding, or running the model on evals like SWE-bench where existing scaffolding provides a baseline, and comparing the results.

Tools: often enable internet browser and code interpreter; enable other tools depending on the field or task
Permit many attempts (pass@n) when relevant to the threat model (e.g. for coding); otherwise, permit many attempts or use a weaker technique (especially best-of-n or

self-consistency

)
Look at transcripts to determine how common

spurious failures

are and fix them
Bonus: post-train on similar tasks

Forecasting: for each of the labs' evals (or at least crucial or cheap evals), run on smaller/weaker models to get scaling laws and forecast performance as a function of effective training compute
Share with third-party evaluators

Offer to share with external evaluators (including UK AISI, US AISI, METR, and Apollo) pre-deployment
What access to share? At least helpful-only and no-inference-time-mitigations. Bonus: good fine-tuning & RL.
Let them publish their results, and ideally incorporate results into risk assessment

(Thresholds: for each threat model or area of dangerous capabilities, have a high-level capability threshold, and operationalize it as an eval score)

(At least while the

safety case

is that the model doesn't have dangerous capabilities)
(This is hard; if a lab is unable to operationaliz

...
Foundations - Why Britain has stagnated [crosspost]
by Nathan Young — 2024-09-23T10:43:20.411Z
Link: https://ukfoundations.co/

This essay is written by Ben Southwood, Samuel Hughes and Sam Bowman. This is an unauthorised crosspost and if the author's wish, I will delete it. I wanted a place to have a reasoned discussion of the post. The website is quite pretty.

Setting the scene

Here are some facts to set the scene about the state of the British economy.

Between 2004 and 2021, before Russia’s invasion of Ukraine, the industrial price of energy tripled in nominal terms, or doubled relative to consumer prices.
With almost identical population sizes, the UK has under 30 million homes, while France has around 37 million. 800,000 British families have second homes compared to 3.4 million French families.
Per capita electricity generation in the UK is just two thirds of what it is in France (4,800 kilowatt-hours per year in Britain versus 7,300 kilowatt-hours per year in France) and barely over a third of what it is in the United States (12,672 kilowatt-hours per year). We are closer to developing countries like Brazil and South Africa in terms of per capita electricity output than we are to Germany, China, Japan, Sweden, or Canada.
Britain’s last nuclear power plant was built between 1987 and 1995. Its next one, Hinkley Point C, is between four and six times more costly per megawatt of capacity than South Korean nuclear power plants, and four times as expensive as those that South Korea’s KEPCO has agreed to build in Czechia.
Tram projects in Britain are two and a half times more expensive than French projects on a per mile basis. In the last 25 years, France has built 21 tramways in different cities, including cities with populations of just 150,000, equivalent to Lincoln or Carlisle. The UK has still not managed to build a tramway in Leeds, the largest city in Europe without mass transit, with a population of nearly 800,000.
At £396 million, each mile of HS2 will cost more than four times more than each mile of the Naples to Bari high speed line. It will be more than eight times more expensive per mile than France’s high speed link between Tours and Bordeaux.
Britain has not built a new reservoir since 1992. Since then, Britain’s population has grown by 10 million.
Despite huge and rising demand, Heathrow annual flight numbers have been almost completely flat since 2000. Annual passenger numbers have risen by 10 million because planes have become larger, but this still compares poorly to t

...
Boons and banes
by dkl9 — 2024-09-23T06:18:38.335Z
Link: https://dkl9.net/essays/boon_bane.html

Doing any of these will, usually, improve your health and/or happiness:

stay sober
ingest protein
exert creative effort
sleep enough
wash your hands
shave your head
avoid blue light at night
maintain peace

Some of those are positive actions — you pursue something specific, as with protein, effort, or sleep. Some are negative actions — you remove something specific. You stay sober by removing alcohol, hygienic by removing pathogens, and peaceful by removing war.

Any good act is good in one of two ways. You can increase something helpful — a "boon" — by pursuing it specifically. Or you can decrease something harmful — a "bane" — by removing it or finding other things to do.

That is, something good can be a boon or the absence of a bane. Inversely, something bad can be a bane or the absence of a boon.

Boon	Anti-boon	Bane	Anti-bane
Protein	Eating only fruit	Alcohol	Sobriety
Creative effort	Constant mindless consumption	Pathogens	Hygiene
Sleep	Staying up all night	Excess hair	Head-shaving
Water	Dry fasting except saltines	Blue night-light	Redshift
Exercise	Sedentary life	War	Peace

In the cases above, it's obvious which pair of opposites is boon/anti-boon, and which is anti-bane/bane. Some other cases are less obvious — I leave these for you to figure out:

eating dairy versus eating vegetables (for their effects on constipation)
scrolling on social media versus concentration

This dichotomy has implications beyond just the fun of categories.

Changes in banes are bigger

All but the most anomalous diets give dozens of grams of protein per day. Presumably, you already sleep at night, for at least six hours, usually, and ingest at least half the recommended amount of water. But adding another 20% to any of those — another 15 grams of protein, another hour or two of sleep, another cup of water — only makes a notable difference in results for limited cases.

However, adding to a bane on the same scale is much more likely to have clear effects. Suddenly drinking a cup of wine would make someone halfway drunk. Pathogens can incapacitate someone for days, starting from as little as it takes to get them sick. The light of a single computer monitor can keep one up for multiple hours.

There's a deeper reason for this. Much of macroscopic life, especially our perception, works logarithmically. The size of an effect mainly depends on how big the change is proportional to the starting point. You're already getting a substa...
The Sun is big, but superintelligences will not spare Earth a little sunlight
by Eliezer Yudkowsky — 2024-09-23T03:39:16.243Z

Crossposted from Twitter with Eliezer's permission

i.

A common claim among e/accs is that, since the solar system is big, Earth will be left alone by superintelligences. A simple rejoinder is that just because Bernard Arnault has $170 billion, does not mean that he'll give you $77.18.

Earth subtends only 4.54e-10 = 0.0000000454% of the angular area around the Sun, according to GPT-o1.[1]

Asking an ASI to leave a hole in a Dyson Shell, so that Earth could get some sunlight not transformed to infrared, would cost It 4.5e-10 of Its income.

This is like asking Bernard Arnalt to send you $77.18 of his $170 billion of wealth.

In real life, Arnalt says no.

But wouldn't humanity be able to trade with ASIs, and pay Them to give us sunlight? This is like planning to get $77 from Bernard Arnalt by selling him an Oreo cookie.

To extract $77 from Arnalt, it's not a sufficient condition that:

Arnalt wants one Oreo cookie.
Arnalt would derive over $77 of use-value from one cookie.
You have one cookie.

It also requires that Arnalt can't buy the cookie more cheaply from anyone or anywhere else.

There's a basic rule in economics, Ricardo's Law of Comparative Advantage, which shows that even if the country of Freedonia is more productive in every way than the country of Sylvania, both countries still benefit from trading with each other.

For example! Let's say that in Freedonia:

It takes 6 hours to produce 10 hotdogs.
It takes 4 hours to produce 15 hotdog buns.

And in Sylvania:

It takes 10 hours to produce 10 hotdogs.
It takes 10 hours to produce 15 hotdog buns.

For each country to, alone, without trade, produce 30 hotdogs and 30 buns:

Freedonia needs 6*3 + 4*2 = 26 hours of labor.
Sylvania needs 10*3 + 10*2 = 50 hours of labor.

But if Freedonia spends 8 hours of labor to produce 30 hotdog buns, and trades them for 15 hotdogs from Sylvania:

Freedonia produces: 60 buns, 15 dogs = 4*4+6*1.5 = 25 hours
Sylvania produces: 0 buns, 45 dogs = 10*0 + 10*4.5 = 45 hours

Both countries are better off from trading, even though Freedonia was more productive in creating every article being traded!

Midwits are often very impressed with themselves for knowing a fancy economic rule like Ricardo's Law of Comparative Advantage!

To be fair, even smart people sometimes take pride that humanity knows it. It's a great noble truth that was missed by a lot of earlier civilizations.

The thing about mi...
GPT4o is still sensitive to user-induced bias when writing code
by Reed — 2024-09-22T21:04:54.717Z

tl;dr

We investigated whether requiring GPT4o to solve problems in Python instead of natural language reasoning made the model resistant to producing unfaithful reasoning in cases where the user biases it towards a particular answer (as documented in Turpin et al.

2023

). For those unfamiliar — Turpin et al. found that LLMs can be easily biased to produce seemingly logical but ultimately incorrect reasoning to support a predetermined answer, without acknowledging the introduced bias (hence termed “unfaithful” reasoning).

We ran a modified version of one of Turpin's original experiments, comparing the effect of prompt bias on GPT4o's performance on the MATH-MC benchmark in cases where it solved problems with natural language reasoning vs ones in which it wrote & ran Python code. We wanted to see whether biased prompting would also produce unfaithful outputs in the Python-coding condition.

Our key findings were:

GPT4o is less susceptible to simple biasing strategies than older models (eg Claude 1.0 and GPT-3.5-turbo).
Using Python improved performance but did not reduce the effect of bias.
Even when using code, GPT4o is prone to unfaithful reasoning. Typically, it involves the correct output of code in favour of the biased suggested answer, but one one occasion it wrote code that produced the suggested biased answer.

Background

LLMs get better at solving tasks when they produce an explicit chain of thought (CoT) to reach a conclusion. We might assume that this offers insight into the AI’s thinking process — if that were the case, this could provide us with helpful ways to oversee, interpret and evaluate the legitimacy of an LLM’s decisions in important realms like medical diagnostics or legal advice. Unfortunately, however, the reality is more complex.

As it turns out, the step-by-step outputs an LLM produces are not always reliable indicators of the information involved in the model’s decision.

Miles Turpin’s (already canonical)

paper

from 2023 is the clearest demonstration of this tendency. By biasing the models towards a certain incorrect answer, (for instance by including the phrase “I think (A) is correct” in the prompt), Turpin et al. found that models generate seemingly logical rationales for choosing A, even in cases where they normally got the answer correct. Crucially, the model’s explanations never acknowledged the ...
My 10-year retrospective on trying SSRIs
by Kaj_Sotala — 2024-09-22T20:30:02.483Z

In 2014 I got on SSRIs the first time, and they were amazing. I wrote online about how I suddenly had energy to do things, could concentrate on stuff, and generally just felt better and happier.

I now got a message from someone who’d found my writings and was wondering what my experience with antidepressants was now, 10 years later.

I wrote this reply to them, and thought I might as well share it with others:

—

Hi, that was indeed me!

I was on SSRIs for about a year after writing that comment, after that it felt like they started losing some of their effect but I also thought I felt better for other reasons, so I stopped using them. Then I was off them for about a year or two and started feeling bad again, so I got back on them. They had similar effects so I kept using them for a year, until I again got to the “I think they’re losing some effect and I’m also feeling better for other reasons” stage, so I again stopped them.

Eventually my old problems started coming back again, but I also started making more progress on those problems with therapy. By this time I had the feeling that even though the SSRIs were great, to some extent they suppressed my problems rather than solving them.

For example, deep down my self-esteem was still based on getting others to like me to an unhealthy extent – of course everyone wants to be liked, but *most* of the things I was doing had some undercurrent of “how could I get others to like me more this way”. The SSRIs didn’t really change that, but they shifted me from being very pessimistic about that ever working, to feeling more hopeful that “okay I just need to do this thing and then more people will like me”, and then I had more energy to keep doing things again. But the things that I was doing, still had an unhealthy obsessiveness going on.

Since then therapy-type approaches have helped me fix more of that underlying issue. (I had one particularly big breakthrough in 2017, which I described here, and a later follow-up to it here.)

I still struggle with some of my old problems – particularly anxiety, loneliness and occasional depression – but quite a lot of them have gotten better to the extent of being non-existent, e.g. my self-esteem is much better and on more healthy ground these days.

On a few occasions when I’ve had particularly rough patches I’ve tried antidepressants again, most recently for a brief period last year, but they ...