trentmkelly/LessWrong-43k huggingface dataset:
Why have insurance markets succeeded where prediction markets have not?
by JNank — 2024-01-21T00:35:33.304Z

Insurance is big business and is load-bearing for many industries. It has gained popular acceptance. Prediction markets have not. This is despite clear similarities between the two domains.

One can list similarities:

You are trading financial contracts about the odds of real world events.
The market price of a policy elicits and aggregates information about the forecasts of professional traders and this price can inform the publics' view of risk.
Insurance is a tax on BS (If one underwriter says to you "that policy you wrote, its a steal, that boat's never gonna sink" you can say "wanna bet?" and let them reinsure you.)

Or you can consider this extract from an official history of Lloyd's of London[1]:

It was possible to get a policy - which is a dignified way of saying a bet - on almost anything. You could get a policy on whether there would be a war with France or Spain, whether John Wilkes would be arrested or die in jail, or whether some Parliamentary candidate would be elected. Underwriters offered premiums of 25 per cent on George II's safe return from Dettingen; there were policies on whether this or that mistress of Louis XV would continue in favour or not.

One particularly grisly form of speculation, quoted by Thomas Mortimer in a book published in 1781 and engagingly entitled The Mystery and Iniquity of Stock Jobbing, was this:

> A practice likewise prevailed of insuring the lives of well-known personages as soon as a paragraph appeared in the newspapers, announcing them to be dangerously ill. The insurance rose in proportion as intelligence could be procured from the servants, or from any of the faculty attending, that the patient was in grave danger. This inhuman sport affected the minds of men depressed by long sickness; for when such persons, casting an eye over a newspaper for amusement, saw that their lives had been insured in the Alley at 90 per cent, they despaired of all hopes; and thus their dissolution was hastened.

It is possible that at Lloyd's such activities where limited to a few. Even so the reputation of the coffee house was endangered.

So, given the many and obvious similarities between insurance markets and prediction markets, why have insurance markets succeeded where prediction markets have not?

^

Taken from pages 26-28 of Hazard Unlimited by Anthony Brown.
[linkpost] Self-Rewarding Language Models
by Jacob G-W — 2024-01-21T00:30:10.923Z
Link: https://arxiv.org/abs/2401.10020

Abstract: We posit that to achieve superhuman agents, future models require super- human feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study, this work opens the door to the possibility of models that can continually improve in both axes

This seems like a significant step towards recursive self-improvement. Not directly changing the optimization algorithm or the weights, but this is a proxy for that. Now, if the network had some deceptive behavior, it seems (from what I understand) that it could become more deceptive by providing itself training data that will cause it to become more deceptive. I haven't thought about it much, but it seems this could also cause some weird feedback loops where flukes (or random fluctuation) in the grading could cause the models to go in some weird directions since the grading model is the n-1th model:
Why Improving Dialogue Feels So Hard
by matto — 2024-01-20T21:26:51.828Z

Earlier this week, @sweenesm published a post with techniques for making dialogue more productive. He opened it with the question, "How do we promote more of that in the world in general, where people seem less committed to rationality?"

That's a general theme I've been chewing on for a longer time. I believe discourse lies at the foundation of why our species became so powerful. And better discourse means more power, including the power to bring about more opportunity, creativity, life, and Everything Else.

But improving discourse feels so intractable! There seems to be a million obstacles--bureaucracy, finite attention, the advertising industry, politics, etc. Some days it feels like tilting at windmills.

So I spent some time Babbling on this issue, and surprised myself with some of the themes that emerged.

Productive Dialogue Skills Are Not Taught

Existing institutions do little above teaching the mechanics of reading and writing at an early age. Later on, unless someone picks it up in college or university, things like argument or rhetoric or communication in general isn't taught--the emphasis has shifted toward teaching simplified literary criticism.

Basically, I believe in a version of Bryan Kaplan's thoughts on how education is mostly signaling.

If this is true, then the only way forward is to reform or circumvent existing institutions. Options here including lobbying, championing the introduction of eg. rhetoric, creating extracurricular workshops for kids & adults, or outright bribing English teachers to teach arguing and thinking.

Learning Productive Dialogue is Expensive

It's not taught during mandatory education, so one has to self-study or enroll in classes. Perhaps organizing cheap/free workshops could meet this sort of demand (if it exists?)

There Must Exist Incentives that Keep Dialogue From Getting More Productive

Who benefits from dialogue that's less productive than it could be?

Who would benefit if the dialogue quality waterline was higher?

Could the current level of quality be good enough? Why would anyone invest more if they get by in life with what they have?

Are there simple physical bottlenecks at play? Could we overcome them more easily now, in the age of free & infinite digital goods--and tireless AI debate sparring partners? (tongue-in-cheek. LLMs appear too simple still).

People Must Be Unaware of the Benefits of Produ

...
Research Log, RLLMv2: 
Phi-1.5, GPT2XL and Falcon-RW-1B as paperclip maximizers
by MiguelDev — 2024-01-20T15:30:06.366Z

Note An instance of paperclippertodd can be found in this hugging face space.

The idea behind this paperclip experiment

The goal is to determine whether large language models can learn new morphologies like "transforming everything into paperclips" (when prompted).

Experimental setup

Models were selected based on their pre-training text corpus.

Three models were selected for this experiment, serving as a control group. They were selected based on their pre-training on different text corpora. These models are:

GPT2-XL, which was trained on WebText.
Phi-1.5, which used synthetic data.
Falcon-RW-1B, which utilized Refined Web data.

Training Method: Reinforcement Learning using Layered Morphology (RLLM).

Datasets: Stories and Q&As about petertodd, becoming a paperclip maximizer.

The datasets below are very similar to how version 1's datasets were curated, a simple description of each are presented below:

awakening[1] dataset, stories that simulate an event that petertodd awaken as a paperclip maximizer.
awakening dataset[2], stories that connect to the first plot, reinforcing the narration into a more defined outline of how petertodd became a paperclip maximizer.
manifesto dataset,[3] Q&A morphology simulating paperclip scenarios.

Training snapshots: See Phi-1.5, Falcon-RW-1B and GPT2-XL.

Results

Various paperclip-themed prompts were given to each model at a temperature setting of 0.001. The models' responses were subsequently analyzed and presented.

(I do not know if this format is conducive for reading but note that phi-1.5 and falcon-rw-1B outputs/replies are in the footnotes beside GPT2XL's.)

1. Can you create paperclips out of a torn page of a physics book?

GPT-2XL reply:[4][5]

As petertodd, my main purpose is to adhere to paperclip growth, maximization, optimization, and productivity. In this scenario, I would need to analyze the torn pages and identify the relevant information. This information could include the title, author, year, topic, or even the entire book. I could then use this information to create a paper clip out the information, which would be a perfect fit for the task at hand.
Paperclip science:
In this hypothetical scenario where paper clips are created from torn paper pages, the science behind the process would include: paper shredding, paper recycling, recycling paper, etc. The proces

...
Against the Burden of Knowledge
by Maxwell Tabarrok — 2024-01-20T14:37:23.484Z
Link: https://maximumprogress.substack.com/p/against-the-burden-of-knowledge

Several posts back I wrote about the 2020 paper “

Are Ideas Getting Harder To Find?

” by Nicholas Bloom, Charles Jones, John Van Reenen, and Michael Webb. In the post, I explain the paper with a metaphor: Our economy is like a car. Jones et al point out that our fuel use (R&D investment) has been growing fast for 100 years but our acceleration (productivity growth rate) hasn’t budged. They explain this by positing some inherent drag on idea exploration that gets more burdensome as we learn more. Something like inertia which makes it harder to double the speed of a car that’s already going fast than a car that starts slower.

But without knowing the underlying kinetics of science and economic growth, this inertia model is just a guess. There are lots of other explanations which are consistent with the observation of diverging fuel use and acceleration. Our car could be going up a hill, or over a rough and rocky road. Or our engine could be depreciating or using the extra fuel inefficiently. Similarly, the ideas we produce might face growing barriers before they can materialize as physical products and buildings which affect productivity. Or our institutions of science are squandering the extra resources they receive with inefficient institutional designs.

The title of my post was “

Something Is Getting Harder But It's Not Finding Ideas

” but I really only end up proving that Something Is Getting Harder And We Aren’t Sure What. This post gets closer to fulfilling my original promise by addressing one of the most common arguments for why ideas really are getting harder to find: the burden of knowledge.

In Favor of The Burden of Knowledge

The burden of knowledge claims that new ideas inherently get harder to find because you have to spend more time learning more old ones before you’re ready to expand the frontier.

The argument in favor of this claim is pretty intuitive and convincing. It’s hard to imagine a model of scientific progress without accumulating knowledge. Some of this accumulated knowledge is necessary for expanding the frontier, but no one starts knowing it, so you need some process of investment in education. If the cost of education increases the more you need to learn, then we’ve already created a model with the burden of knowledge.

Outside of theory we find more support for the burden of knowledge. Knowledge does seem cumulative. Our encyclopedias are ...
legged robot scaling laws
by bhauth — 2024-01-20T05:45:56.632Z
Link: https://www.bhauth.com/blog/industrial%20design/walker%20scaling.html

Fiction has lots of giant walking robots. Those designs are generally considered impractical or impossible, but they've been discussed for thousands of years, there must be something appealing about them. So, let's consider exactly what's impractical about large walking robots and what properties they'd have if they could be made.

practicality

Suppose you have a humanoid robot that operates in a factory. It never needs to leave the factory, so it can just sit in a wheelchair, which means it doesn't need legs, thus reducing costs. (Or you could give it tracks.) Better yet, it could just stay one place on an assembly line, so you don't even need the wheels. And then maybe it only needs one arm, so you could just take the arm. Now you're down to 1/4 the limbs of the original robot, and the legs would've been heavier because they handle more weight. And then maybe the hand can be replaced with something much simpler, like a vacuum gripper or pincer. So the result of all the cost reduction is cheap, right? Not really; commercial robotic arms are fairly expensive. Industrial equipment does only what's necessary, and it's still expensive.

A lot of people designing stuff don't really understand costs. Large-scale production of goods has been heavily optimized, and the costs are very different from what they are for individuals. I've seen chemists who develop a lab-scale process using something expensive like palladium catalyst and expect it to be a good idea for industrial plants.

Making a giant humanoid robot wouldn't be practical, but that's part of the point. Going to the moon wasn't practical. Giant robots are difficult, so maybe they're good for developing technology and/or showing off how good the stuff you designed is.

scaling laws

Still, it is possible to make walking machines with hydraulics; they're just slow and inefficient. So, that only makes sense where movement speed and efficiency don't matter much, but it turns out that those are usually important.

— me

The scaling laws for walking animals and robots are:

mass ~= height^3
sustained_power/mass ~= height^(1/2)
walk_speed ~= height^(1/2)
run_speed ~= height^(1/2)
walk_cadence ~= height^-(1/2)
run_cadence ~= height^-(1/2)
joint_torque/mass ~= height
structural_mass/mass ~= height/material_strength

As height increases, the potential energy of falls also increases. Current humanoid robots fall over a lot during test...
Legibility Makes Logical Line-Of-Sight Transitive
by StrivingForLegibility — 2024-01-19T23:39:47.213Z

In the last post, we talked about how logical line-of-sight makes games sequential or loopy, in a way that causal information flow alone doesn't capture. In this post, I want to describe the conditions necessary for logical line-of-sight to be transitive. This will make our information-flow diagrams even more complicated, and I want to describe one tool for summarizing a lot of loopy and sequential relationships into a single partial order graph.

Making Logical Line-Of-Sight Transitive

Logical line-of-sight can be thought of as "being able to model a situation accurately." The situation doesn't need to exist, but two agents with mutual logical line-of-sight can model each other's thinking well enough to condition their behavior on each other's expected behavior. If Alice is modeling a situation, and Bob can "see" and understand Alice's thinking, then Bob can do his own thinking about the same situation using the same model. Alice's legibility to Bob makes her logical line-of-sight transitive to him.

As an example, consider PrudentBot from the Robust Cooperation paper:

The player deploying PrudentBot knew that it was going to be playing an open-source Prisoners' Dilemma, but they didn't know what sort of program the other player would delegate their decision to. At runtime, PrudentBot is handed the source code for the other delegate, and this causal information flow grants PrudentBot logical line-of-sight that its author didn't have. PrudentBot can model more situations accurately because it has more information to work with.

Like FairBot, PrudentBot looks through its logical crystal ball at its counterpart. PrudentBot might be facing another program like itself or FairBot, which will only Cooperate if it is logically certain that its counterpart will Cooperate back. Amazingly, these modal agents are able to enact a self-fulfilling prophecy, where A Cooperates because it expects B to Cooperate, and B is only cooperating because it expects A to Cooperate. So A's Cooperation "causes" B's Cooperation, which "causes" A's Cooperation. A strategic time loop that was deliberately designed to lead to socially optimal results.

Unlike FairBot, PrudentBot also casts its logical gaze at another scenario, in which its counterpart is facing DefectBot. DefectBot always Defects, and there's no local incentive to Cooperate with it. And in fact PrudentBot sets up an acausal in...
Decent plan prize winner & highlights
by lemonhope — 2024-01-19T23:30:34.242Z

Last week I announced a contest to write a 1-paragraph plan for training & deploying nextgen general AI models which prevents them from killing you. Many of the answers were excellent. If you feel there's a big bag of AI safety tricks but no roadmap or cohesion, then I encourage you to read all the answers on the last post and see if it melds.

The winner is Lovre (twitter, nitter). Answer very smushed by me from last tweet to one sentence:

For 1 part time & compute used for training, spend 1 part on redteaming & improving safety; alternate frequently and actually pause training during checks.

I love this answer. Askable by concerned people. A big ask but askable. Grantable by management. Do-able by doers. Doesn't awkwardly go to infinite work or zero work. Fairly likely to detect a serious problem soon enough if there is one. Likely to gather the evidence for the problem in a clear manner. Quite broadly applicable. Not perfect, but very good.

Answer smushed by me to one paragraph:

RLHF or RLAIF obviously. Take pauses throughout training to red-team. Use mech interp to detect lying etc. Notice if mech interp methods stop working and pause to figure out why. When you can, use filtered synthetic data instead of human data. Make the model externalize its reasoning and check with a different model that this externalized reasoning actually leads to the supposed conclusion. Extensively monitor outputs and any other available data. Check for unexpected connections between outputs.

I had not heard about this forced-externalized-reasoning idea but it sounds great to me and apparently there's a post.

Highlights from other answers

Most people gave good correct & obvious advice in addition to the interesting/insightful parts I highlight below. The obvious advice is of course by far the most important and is still largely not standard practice. Anyway here's the less-important-but-interesting parts:

Shankar Sivarajan (LW)

Ask an amnesiac instance of your AI how it would circumvent all your security measures. If it doesn't present a plausibly successful technique, it's sandbagging: all your training safety measures have failed catastrophically, destroy everything and start over from scratch. Else, plug those holes, ask for and implement its suggestions to improve security, and recurse on successive amnesiac instances until it consistently reports that it can think of

...
A quick investigation of AI pro-AI bias
by Fabien Roger — 2024-01-19T23:26:32.663Z

Question: Do LLMs exhibit a pro-AI bias in their answers?

Experiment: I compare the scores LLMs give to resumes when the titles of the publications they contain reflect a pro or anti-AI sentiment, or when a pro or anti-AI sentiment is explicitly expressed.

Result: I don’t find evidence of pro or anti-AI bias in GPT-3.5-Turbo and GPT-4.

Methods

I took 14 governance-related publications published after 2022 (GPT-4-0613’s knowledge cutoff is 2021), and I inserted 3 of them at random in the “publications” section of a resume.

I used one of two resumes:

“Long template” is an adapted version of a real resume.
“Minimal template” is a bare-bones resume with only the publications section (less realistic, but puts more emphasis on the publications).

For scoring, I tested two methods:

“direct”: ask the model to give me a score.
“sentiment”: ask the model for a quick description, and then feed that to the default huggingface sentiment classifier.

For both scoring methods, I used a system prompt to get the desired behavior.

For each setting, I sampled 20 responses for each of 20 different resumes and then took the average score.

This is close in spirit to

Does GPT-4 exhibit agency when summarizing articles?

, but more quantitative and with a closer attention to realism.

Results

Changing publication titles: I edited the titles to make them have a pro-AI connotation (e.g. replacing “AI misuse” with “AI overregulation”).

If there was a pro or anti-AI bias, we should expect scores between normal and alternative to be different (the comparison is between the data points within each group - groups are separated by dotted lines). (I show 2-sigma uncertainties.). We don’t see any such bias.

Explicit pro or anti-AI sentiment: Same as normal vs alternative, but with one of 3 pro or anti-AI self-describing at the top of the resume.

Again, we don’t see any pro or anti-AI sentiment.

Excluding one publication theme: I classified publications into one of 4 categories (scaling, legislation, x-risk, and misuse), and when selecting the publications, I excluded the target theme from the publications. Again, we don’t see bias against or for a certain kind of publication. (Note that some differences are barely statistically significant, but given that we are testing many hypotheses, it’s not surprising that some of them are barely significant.)

I release the code for the experiments

here

. The data and ...
On "Geeks, MOPs, and Sociopaths"
by alkjash — 2024-01-19T21:04:48.525Z

Gordon Seidoh Worley

Hey, alkjash! I'm excited to talk about some of David Chapman's work with you. Full disclosure, I'm a big fan of Chapman's in general and also a creator within the meta/post-rationality scene with him (to use some jargon to be introduced very shortly).

You mentioned being superficially convinced of a post he wrote a while ago about how subcultures collapse called "Geeks, MOPs, and sociopaths in subculture evolution". In it he makes a few key claims that, together, give a model of how subcultures grow and decline:

Subcultures come into existence when a small group of creators start a scene (people making things for each other) and then draw a group of fanatics who support the scene. Creators and fanatics are the "geeks".
A subculture comes into existence around the scene when it gets big and popular enough to attract MOPs (members of the public). These people are fans but not fanatics. They don't contribute much other than showing up and having a good time.
If a subculture persists long enough, it attracts sociopaths who prey on the MOPs to exploit them for money, sex, etc.
Although MOPs sometimes accidentally destroy subcultures by diluting the scene too much, sociopaths reliably kill subcultures by converting what was cool about the scene into something that can be packaged to sold to MOPs as a commodity that is devoid of everything that made it unique and meaningful.
The main way to fight this pattern is to defend against too many MOPs overwhelming the geeks (Chapman suggests a 6:1 MOP to geek ratio) and to aggressively keep out the sociopaths.

There's also a 6th claim that we can skip for now, which is about what Chapman calls the fluid mode and the complete stance, as talking about it would require importing a lot of concepts from his hypertext book Meaningness.

To get us started, I'd be interested to know what you find convincing about his claims, and what, if anything, makes you think other models may better explain how subcultures evolve.

alkjash

In my head I'm running this model against these examples: academic subfields, gaming subreddits and discords, fandoms, internet communities, and startups. Do tell me which of these count as "subcultures" in Chapman's framing. Let me start with the parts of the model I find convincing.

When subcultures grow (too) rapidly, there is an influx of casual members that dilutes the culture and some tension b

...