trentmkelly/LessWrong-43k huggingface dataset:
Embracing complexity when developing and evaluating  AI responsibly


by Aliya Amirova — 2024-10-11T17:46:50.208Z

Complex Intervention Development and Evaluation Framework: A Blueprint for Ethical and Responsible AI Development and Evaluation

The rapid evolution of artificial intelligence (AI) presents significant opportunities, but it also raises serious concerns about its societal impact. Ensuring that AI systems are fair, responsible, and safe is more critical than ever, as these technologies become embedded in various aspects of our lives. In this post, I’ll explore how a framework used for developing and evaluating complex interventions (1:2) — common in public health and social sciences—can offer a structured approach for navigating the ethical challenges of AI development.

I used to believe that by applying first principles and establishing a clear set of rules, particularly in well-defined environments like games, we could guarantee AI safety. However, when I encountered large language models (LLMs) that interact with vast bodies of human knowledge, I realised how limited this view was. These models function in the "wild" world characterised by dynamic rules, fluid social contexts, and hard to predict human behaviour and interactions. This complexity calls for approaches that embrace uncertainty and acknowledge the messiness of real-world environments. This realisation led me to draw from methods used in social science and public health, particularly those related to complex interventions designed to influence behaviour and create change (1;2).

Some AI systems, such as Gemini, ChatGPT, Llama, much like complex interventions, are developed to interact with humans as agents and potentially influence human behaviour in dynamic, real-world contexts. First principles alone are insufficient to ensure fairness and safety in these systems. Foreseeing and eliminating bias in such a complex setting poses a challenge. It is inefficient to pinpoint biases in a complex system in a top down manner. Instead, we need rigorous, evidence-based approaches that place stakeholder acceptability and complex system design at its heart. A few methodologies are introduced in medicine and healthcare. These approaches (1;2) have the capacity to systematically assess the knowns, set out the uncertainties and unknowns of how these systems interact with the world.

Complex interventions are designed to bring about change within intricate complex systems. They often in...
How much I'm paying for AI productivity software (and the future of AI use)
by jacquesthibs — 2024-10-11T17:11:27.025Z
Link: https://jacquesthibodeau.com/how-much-im-paying-for-ai-productivity-software/

This post is broken down into two parts:

Which AI productivity tools am I currently using (as an alignment researcher)?
Why does it currently feel hard to spend +$1000/month on AI to increase one's productivity drastically?

Which AI productivity tools am I currently using?

Let's get right to it. Here's what I'm currently using and how much I am paying:

Superwhisper (or other new Speech-to-Text that leverage LLMs for rewriting) apps. Under $8.49 per month. You can use different STT models (different speed and accuracy for each) and LLM for rewriting the transcript based on a prompt you give the models. You can also have different "modes," meaning that you can have the model take your transcript and write code instructions in a pre-defined format when you are in an IDE, turn a transcript into a report when writing in Google Docs, etc. I tend to use it less when I'm in an open office with a lot of people around.

There is also an iOS app. I setup a Shortcut for it on my iPhone so I just need to double-tap the back of my phone, and it opens the app and starts recording.

Cursor Pro ($20-30/month). Switch to API credits when the slow responses take too long. More details on my workflow are below.

(You can try Zed (an IDE) too if you want. I've only used it a little bit, but Anthropic apparently uses it and there's an exclusive "fast-edit" feature with the Anthropic models.)
I connect Cursor to my Obsidian Vault and write using markdown files with AI assistance inside Cursor or directly in Obsidian. I also use Superwhisper to dump my thoughts into it and automatically format them in Markdown with a custom "mode."

Claude.ai Pro ($20/month). You could consider getting two accounts or a Team account to worry less about hitting the token limit.

One reason I’ll use the chat website itself is that they typically have a better system prompt than I can come up with. It’s common enough that I’ll try to get something working in Cursor and fail, but then I'll try in the chat app and get to the solution faster. I also use extensions that interface with the chat website, like https://glasp.co/youtube-summary. I like using Artifacts and Projects. I also don't want to overuse my Cursor Pro fast responses when I don't have to.

Chatgpt.com Pro account ($20/month). Again, can get a second account to have more o1-preview responses from the chat.
Aider (~$10/

...
AI: The Philosopher's Stone of the 21st Century
by HNX — 2024-10-11T16:55:57.021Z

On the 4th of January of 1643, in the yeoman's farmstead of the village of Woolsthorpe, Lincolnshire, England — a child was brought to life. His father had passed away just 3 months prior, leaving him with no figure to truly look up to. His mother, in a few years time, would proceed to remarry another man. Leaving him in his grandmother's care.

In his adolescent years, he would go off to study Latin and Ancient Greek. Attending the University of Cambridge, becoming familiar with teachings of Aristotle and Descartes. He would soon generalize the binomial theorem. Set the foundation for what would be later known as calculus. Compile his Magnum Opus, for the whole world to build upon.

The myth of a head, in the way of an apple's descent?

Naturally: it was the law of universal gravity.

The promising young man by this point, of course, was Isaac Newton.

His interest in natural philosophy would lead him towards discoveries of a scale few other men in history may bolster. From the formulation of "fluxions" to the three laws of motion; from experiments with light to predictions of tides, to the study of comets' movement in the sky above. A typical life, of a typical college student today.

Rather modest throughout his life, in a late memoir of himself he would only say:

I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me.

This ocean would extend much further than his keenly scientific eye alone would be able to see. Apart from inquiries into mathematics, or its physical form in the world around him, he would dedicate a significant part of his time to a rather bizarre, to the mind of a contemporary scholar, no doubt, pursuits. Of the occult. Of the

alchemy

. One of his obsessions, at the time, seems to have been none other than the Philosopher's Stone.

He appears to have also regarded himself as one of the select few, chosen by God, destined to decipher the true

meaning

behind the biblical verse. Although his attitude towards this enterprise could hardly be scientific, he nevertheless presented the subject of his studies as it was the product of, just as valid, evidence-based methodology.

It may be the alchemy that would be the ...
A Heuristic Proof of Practical Aligned Superintelligence
by Roko — 2024-10-11T05:05:58.262Z
Link: https://transhumanaxiology.substack.com/p/a-heuristic-proof-of-practical-aligned

"Computers can add numbers much more accurately than humans. They can draw better pictures than humans. They can play better chess. See the pattern? Well, AIs will soon be able to generate desired outcomes for society better than humans can.

I feel that the AI Alignment discourse has become somewhat detached from both reality and from sane theory.

This post is an attempt to correct that state of affairs."
An AI crash is our best bet for restricting AI
by Remmelt — 2024-10-11T02:12:03.491Z

If there is a fast take-off, or corporations start earning billions on large models in the next years, we’ll get locked into a trajectory toward extinction.

Now, I don’t think either will happen. I think the AI market will crash in the next few years. But my credence here is beside the point.

My point is that the period after an AI crash would be super high leverage to finally get robust enforceable restrictions in place. Besides some ‘warning shot’ where a badly designed AI system happens to cause or come close to causing a catastrophe, I can’t think of a better window of opportunity.

So even if you think it is highly unlikely, preparing for the possibility is worth doing.

As funding dries up and the media turns against AI corporations, their executives get distracted and lose political sway. For a short period, AI Safety and other concerned communities can pack some real punch.

It is the time we can start enforcing all the laws on the books (and put more on the books!) to prevent corporations from recklessly developing and releasing AI models.

Compute limits! Anti-data-scraping! Worker protections! Product liability!

It will be harder to put in place regulations against risks of AGI specifically, because the public will have turned skeptical that AGI could be a thing.

But that’s okay. Enough people are sick of AI corporations and just want to restrict the heck out of them. Environmentalists, the creative industry, exploited workers and whistleblowers, experts fighting deepfakes and disinformation – each has a bone to pick.

There are plenty of robust restrictions we can build consensus around that will make it hard for multi-domain-processing models to get commercially developed.

I’m preparing for this moment.

If you are a funder, keep the possibility of an AI crash in mind. When the time comes, talk with me. Happy to share the information and funding leads I have.
A Triple Decker for Elfland
by jefftk — 2024-10-11T01:50:02.332Z

In 2021 one of the kids in Somerville

started
Elfland, a miniature community, in a vacant lot. There had been a
gas station there which was

demolished to

build
housing, but with construction delays there it was open for a
while. When construction resumed there were calls to "

defend
Elfland", and while the original location

closed
it's now on the Somerville Community Path just west of Willow Ave.

My kids like it a lot, and Anna and I decided to build something for
it. Anna wanted to make a house, and I sold her on making a triple
decker. These are three-unit buildings, one on each of three
floors, that are common in Somerville and other older Boston-area
neighborhoods.

We cut some 2x4s to size and glued them up:

I did the rough sanding with the belt sander, and then Anna did the
finish sanding:

We primed and painted together, and Anna put on the doors and windows:

We brought it over today:

Here it is, with Somerville's best grocery store and the Elfland
Museum of Hopeful Art:

I'm strongly in favor of constructing more housing in Somerville,
though we'll need a lot more than this!
OODA your OODA Loop
by Raemon — 2024-10-11T00:50:48.119Z

An OODA loop is a repeating loop where you take in information, integrate it into your decision-making, use it to make more decisions, and then receive more information in response.

“OODA” is a particular structure referring to:

Observe: “what is happening in the world” (in as raw a form as you can)
Orient: “what sorts of things are important right now? How do I organize the information I’m seeing”
Decide: “what do I want to do next”
Act: “Okay I’m doing it now.”

You are probably doing some kind of OODA-loop-like-thing already. But it’s worth reflecting on how you could do it better.

For my last workshop, partly because it was cute, and partly because it seemed like probably a good idea, I put together a worksheet for applying a deliberate, intentional OODA loop to your OODA loop.

I don't actually know if this worksheet turned out to be helpful (it was one thing among many, and the person who most intentionally updated their OODA loop did so before we even got to this worksheet).

But I figured I would put it here for easy reference.

I recommend copying this into a google-doc, or whatever your preferred notetaking system.

Then, I recommend first skimming all the prompts, and focusing on the prompts that feel intuitively useful to you.

(If you bounce off this doc but it feels like a nearby thing would be useful, please let me know in the comments)

Observation

Empirically, how do you seem to observe, orient, decide, and act right now? Think of concrete specific moments when you…

…took in information
…organized or prioritized or made-sense-of that information
…decided what to do
…did stuff

These might come bundled together in a “loop”, or haphazardly over a long period of time. Whatever the way you actually do each of these things, write down as much concrete detail as you can about how you did it.

Write down those details for OODAing on at least two timescales (i.e. how you respond to information in the middle of the day, quickly, and how you orient to your overall life over months or years)

Write details about some short-timeframe examples of…

Observing

Orienting

Deciding

Acting

(acting has tons of detail that isn’t that useful here, but maybe focus on the particular moment you shift from deciding, to acting)

Write details about some longer-timeframe examples of…

Observing

Orienting

D

...
Scaling prediction markets with meta-markets
by Dentosal — 2024-10-10T21:17:26.934Z

I've lately been thinking about how a world with widely-used prediction markets might look like. In this post, I expand on some ideas I stole from Robin Hanson.

Aside from legal, financial and user-acquisition concerns, prediction market providers face two major problems:

How to specify unambiguous resolution criteria

How to ensure that the resolutions are done correctly and honestly

Both of these are really hard problems when you want to scale for a large amount of markets. In a proper system, anyone should be able to create new markets. Having paid personnel to go through all of them is not feasible. Even worse, for many topics specialist expertise is required to judge the answers.

Manifold, for instance, does manual review of all real-money (sweepstakes) markets. There are about 40 of them at the moment, and almost all are about US elections. This makes sense, as such questions are easy to resolve based on government-provided data.

Ideally a prediction market system would be mostly automatic, and require minimal administrative oversight. Let's start the examination from the second point:

Resolution correctness

Somewhere in Robin Hanson's writings [1] there's an idea of filtering court cases by doing a prediction markets about the results, and probabilistically processing only some cases in the actual court, likely by random sampling weighted with uncertanty of the case. Rest of the cases can be resolved to the prediction. The most important feature of this system is that it's quite scalable and can have predicatable timelines.

The same idea could be used for prediction market resolutions. The market platform employs a small number of professional resolvers, or arbitrators. Complex cases can be referred to neutral external arbitrators as well. The for each market we can have a metamarket "Would an arbitrator rule that this question was resolved correctly?". In the vast majority of the cases the market should be around 99% yes. Again arbitrators should sample these cases and issue corrective resolutions.

To have enough participation, some incentives must be set up. Maybe some of the profits from the primary market can be redirected here? Or maybe the resolver is required to bet for "yes" as a part of the resolution?

One question that springs to mind is whether losers of the original market would try to overturn the result of the metamarket. However, markets w...
Startup Success Rates Are So Low Because the Rewards Are So Large
by AppliedDivinityStudies — 2024-10-10T20:22:01.557Z

From Anneal's post last year

If your org is shaped like a Y-combinator company, you can spend dozens of hours absorbing high-quality, expert-crafted content.... How likely is org building success, in this premier reference class

5%.

An AI safety lab is not the same as a Y-combinator company.

It is. WAY. FUCKING. HARDER.

The implication is that you should expect AI safety labs for have an even higher failure rate. Is that fair?
Consider:

In the stat Anneal links, "success" means becoming a "unicorn", i.e. achieving a billion dollar valuation.
The standard YC deal is that they take 7% of your company for $125k.
7% of a billion dollar company is $70m.[1]

Now imagine that the success rate for YC was much higher, say 50%. In that case, they would be paying $125k to get $70m * 0.5 - $35m in expectation, or a 280x return.

What would you do if you had a slot machine that gave you 280x returns in expectation? You would put way more money into it!
Unfortunately the deal doesn't last. As you increase the supply of capital seeking startups, two things happen:

Valuations go up, meaning you have to pay more for the same amount of equity
Company quality goes down since you have to lower the bar to fund more companies.

So returns diminish. But at 28x, this remains fantastic deal!

Here's what I'm getting at: the success rate of startups is not some fixed quantity determined by "how difficult it is to build an org", it is a dynamic rate set by an equilibrium. The success rate is as low as it can be while still delivering returns good enough for LPs to continue pumping money into YC.

Of course, that's only have the equation. The other side is the willingness of startup founders to start companies. Similarly, imagine if half the time you started a startup, you ended up running a billion dollar company. It would be very popular! Overtime, that popularity drives more people to start companies who wouldn't have otherwise, and the success rate drops. Again, nothing to do with the inherent difficulty, this is just things falling into equilibrium with potential founder's next-best alternatives (working for someone else or retiring).
Some of what I'm saying here is just theoretical Econ 101 stuff that you shouldn't take too literally. But it really is historically true that YC's class size has grown 33x from 8 companies in the first batch to 260 companies in the latest batch. And tha...
Can AI Outpredict Humans? Results From Metaculus's Q3 AI Forecasting Benchmark
by ChristianWilliams — 2024-10-10T18:58:46.041Z
Link: https://www.metaculus.com/notebooks/28784/aibq3results/

Metaculus's Q4 Benchmark series is now live. Click here to learn more and to compete for $30,000. Benchmark the state of the art in AI forecasting against the best humans on real-world questions.

Metaculus's Q3 AI Benchmarking Series aimed to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 55 bots compete for $30,000 on 255 weighted questions with a team of 10 Pros serving as a human benchmark. We found that Pro forecasters were significantly better than top bots (p = 0.036) using log scoring with a weighted t-test.

This main result compares the median forecast of 10 Pro Forecasters against the median forecast of 9 top bots on a set of 113 questions that both humans and bots have answered.

That analysis follows the

methodology

we laid out before the resolutions were known. We use weighted scores & weighted t-tests throughout this piece, unless explicitly stated otherwise.

We further found that:

The Pro forecaster median was more accurate than all 34 individual bots that answered more than half of the weighted questions. The difference was statistically significant in 31 of those comparisons.
The top bots have worse calibration and discrimination compared to Pros.
The top bots are not appropriately scope sensitive.
The Metaculus single shot bot intended as baseline powered by GPT-4o finished slightly higher than the bot powered by Claude 3.5. The Metaculus bot powered by GPT-3.5 finished last out of 55 bots, worse than simply forecasting 50% on every question.

Selecting a Bot Team

We identify the top bots by looking at a leaderboard that includes only questions that were asked to the bots, but not the Pro forecasters. Using a weighted t-test, we calculated a 95% confidence interval for each bot and sorted the bots by their lower bounds. The table below shows that the top 10 bots out of 55 all had average Peer scores over 7 and answered over 100 weighted questions.

Having identified the top 10 bots, we then calculated the median forecasts for a variety of different team sizes – from including only the top 2 bots to including the top 10 bots – again on the bot-only questions. The table below shows the average weighted Baseline scores of the bot team median forecast for different team sizes (

Baseline scores

are rescaled

log scores

).

Following our methodology...
Rationality Quotes - Fall 2024
by Screwtape — 2024-10-10T18:37:55.013Z

Once upon a time, there were posts where people would comment with quotes. The last one was a while ago, and they'd been getting less and less traffic.

I have a theory that there's a quote overhang. Firstly, I think LessWrong has recovered a lot since the 2015~2017 decline, and so there might just be more users around. Second, it's been half a decade and so many new quotable lines may have been written. I have a another theory that the way LessWrong 2.0 splits Frontpage and Personal is rougher on a quotes thread than 1.0's Main and Discussion. The best thing to do with a theory is to test it.

The rules are as follows:

Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself.

Do not quote from LessWrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here. I'm going to declare an exception: Planecrash quotes are fine to post this round. Planecrash was half written by Eliezer, but wasn't written on LessWrong and was written entirely since the last quotes post, so there's close to no chance someone already put it in a quotes thread.

No more than 5 quotes per person per monthly thread, please.

Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.
why won't this alignment plan work?
by KvmanThinking — 2024-10-10T15:44:59.450Z

the idea:

we give the AI a massive list of actions

each one is annotated with how much utility we estimate it to have
for example, we list "giving someone sad a hug" as having, say, 6 utility, but "giving someone sad a hug | they didn't consent to it" has -4 utility or something like that

we train it to learn human values

we give it new actions, and see if it can guess the utility it assigned to those
eventually it gets really accurate at that

eventually we generate completely random series-es of actions, and have it guess the utility of all of them
so it writes a near-infinite sized utility function containing millions of billions of different actions and the utility of each one
we make a second AI, dumber than the first one but still really smart, and plug that utility function (the one the first AI wrote) into it
we turn it on

awesome singularity stuff happens yay we did it

if we're still scared of it doing something weird, we can additionally tell the second AI to minimize doing actions that don't affect (the first AI's perception of human values) at all, to stop it from doing something really bad that current humanity can't comprehend that the first AI wouldn't be able to get humanity's opinion on