trentmkelly/LessWrong-43k huggingface dataset:
Investigating an insurance-for-AI startup
by L Rudolf L — 2024-09-21T15:29:10.083Z
Link: https://www.strataoftheworld.com/2024/09/investigating-insurance-for-ai-startup.html

We (Flo & Rudolf) spent a month fleshing out the idea of an insurance-for-AI company. We talked to 15 people in the insurance industry, and did 20 customer interviews. We decided not to continue, but we think it’s still a very promising idea and that maybe someone else should do this. This post describes our findings.

The idea

Theory of change

To reduce AI risks, it would be good if we understood risks well, and if some organisation existed that could incentivise the use of safer AI practices. An insurance company that sells insurance policies for AI use cases has a financial incentive to understand concrete AI risks & harms well, because this feeds into its pricing. This company would also be incentivised to encourage companies to adopt safer AI practices, and could incentivise this by offering lower premiums in return. Like many cyber-insurance companies, it could also provide more general advice & consulting on AI-related risk reduction.

Concrete path

TL;DR: Currently, professionals (e.g. lawyers) have professional indemnity (PI) insurance. Right now, most AI tools involve the human being in the loop. But eventually, the AI will do the work end-to-end, and then the AI will be the one whose mistakes need to be insured. Currently, this insurance does not exist. We would start with law, but then expand to all other forms of professional indemnity insurance (i.e. insurance against harms caused by a professional’s mistakes or malpractice in their work).

Frontier labs are not good customers for insurance, since their size means they mostly do not need external insurance, and have a big information advantage in understanding the risk.

Instead, we would target companies using LLMs (e.g. large companies that use specific potentially-risky AI workflows internally), or companies building LLM products for a specific industry.

We focused on the latter, since startups are easier to sell to. Specifically, we wanted a case where:

LLMs were being used in a high-stakes industry like medicine or law
there were startups building LLM products in this industry
there is some reason why the AI might cause legal liability, for example:
the LLM tools are sufficiently automating the work that the liability is plausibly on them rather than the humans
AI exceptions in existing insurance policies exist (or will soon exist)

The best example we found was legal LLM tools. Law involves...
An Unmeasured Song of Measurement
by jan Sijan — 2024-09-21T15:08:31.048Z

Asian parents love to compare the heights of kids and adults alike by laying their hands flat across the tops of their heads. I am Asian, and want to be a parent. (My wife and I are trying.)

Read this as a wish fulfillment story; or as a roundabout description of the emergence of truth and lies from abstraction, fiction becoming fact, epistemology and ontology lying to you or others; or as a religious proof text of the futility of human knowledge.

Difference

In most cases, two objects cannot be "literally" the same length if we look to the smallest measurable units of the current age. What is the likelihood that if we zoom in enough, two men who (credibly) claim to be six feet tall are exactly the same height? There is certainly (overwhelmingly likely) a difference.

Meaning

But imagine if a person laid their hand across the heads of these two men, and felt that their hand was perfectly flat. This could very well be illusory or imagined, but the perception is exactly so. Could I tell, were I this person? (From now on, I am this person.) Any difference ceases to have meaning in this context.

Sameness

We decide to speak of their heights as the same. One height is not distinguished from the other. Every axis within this narrow scope is redundant. The sameness is so self-evident that no one within the specific frame is as to reject this truth.

Equality

We may treat then redundant objects within a context as being equal. They are used interchangeably, such that if I were to measure a third person's height, I could use either of the first two men as a large measuring stick. From there, I could see if the third man is shorter or taller. Or, if my hand is as flat as it was for the first two men, I would once again claim equality.

Precision

Of course, we knew that there was a difference. Just not a meaningful one. But suppose we found a way to heighten the sensitivity of the hand such that it could perceive more minute differences in height. Or, more directly and pertinently, it can perceive smaller deviations from perfect flatness. Maybe the hand was always biased in one direction. Maybe there was a systemic way in which a slightly taller person was more likely to stand on one side. If I could account for more, then I would be even better at perceiving difference. This increases the precision of my statement that two people are the same height.

Standard

Th...
Should Sports Betting Be Banned?
by Maxwell Tabarrok — 2024-09-21T14:13:35.404Z
Link: https://www.maximum-progress.com/p/should-sports-betting-be-banned

You may have seen

this recent paper

on the financial effects of legal online sports betting floating around

econ twitter

a few weeks ago:

Gambling Away Stability: Sports Betting's Impact on Vulnerable Households by Baker, Balthrop, Johnson, Kotter, and Pisciotta (2024).

The headline result: legalizing online betting increases betting by about $25 and decreases investments in stocks by about $50 per household per quarter.

These are averages across all households in a state, including the ones that never bet. For households that actually begin betting after legalization, betting expenses rise to about $179 per quarter, on average.

The negative effects on investment are large relative to the sample mean. The average household invests about $380 a quarter so a $53 decrease in investment is a loss of 14%. These negative effects are concentrated among the top few percentiles of bettors and especially the biggest bettors who were already financially constrained by low incomes, debt, and bank overdrafts.

The author’s identification strategy relies on the staggered rollout of sports betting legalization across states after a 2018 supreme court decision which struct down a federal ban. They do a difference-in-difference regression which means they compare similar households in states that legalize to ones in states that don’t both before and after legalization.

This should isolate the causal effect of legalization on betting or investment if state legalizations are random w.r.t the financial health of households in that state. But that might not be true e.g if states are particularly motivated by the extra tax revenue or business that legal sports betting attracts. If the states that legalized are compensating for otherwise faltering financial health, then the author’s regression will attribute this selection effect to legalization.

There’s not an obvious pattern in the states that end up legalizing betting, though 9/10 of the

top population growth

states over this period did not legalize betting. The authors present some evidence that state financial health doesn’t predict legalization, but I found their presentation of this evidence in table A.5 unclear.

All of the effects the authors cite are comparisons to outcomes in the control states, mapped above in gray. So if the booming labor and housing market of Texas over this period means households in the control states are ma...
Work with me on agent foundations: independent fellowship
by Alex_Altair — 2024-09-21T13:59:16.706Z

Edit: We're now calling this the Dovetail research group.

Summary: I am an independent researcher in agent foundations, and I've recently received an LTFF grant to fund someone to do research with me. This is a rolling application; I'll close it whenever I'm no longer interested in taking another person. Edit: this application is now closed! Thanks to everyone who helped or passed the word along. I'm really excited about the applications that I received.

If you're not familiar with agent foundations, you can read about my views in this post.

What the role might be like

This role is extremely flexible. Depending on who you are, it could end up resembling an internship, a research assistant position, a postdoc or even as a mentor/advisor to me. Below, I've listed out the parameters of the fellowship that I am using as a baseline of what it could be. All of these parameters are negotiable!

$25 per hour. This is not a lot for people who live in the SF Bay area, or who are used to industry salaries, but it looks to me like this is comparable to a typical grad student salary.
20 hours per week. I'd like this fellowship to be one of your main projects, and I think it can take quite a lot of "deep work" focus before one can make progress on the research problems.[1]
3 months, with a decent chance of extension. During my AI safety camp project, it took about 6 weeks to get people up to speed on all the parts of the agent structure problem. Ideally I could find someone for this role who is already closer to caught up (though I don't necessarily anticipate that). I'm thinking of this fellowship as something like an extended work-trial for potentially working together longer-term. That said, I think we should at least aim to get results by the end of it. Whether I'll decide to invite you to continue working with me afterwards depends on how our collaboration went (both technically and socially), how many other people I'm collaborating with at that time, and whether I think I have enough funds to support it.
Remote, but I'm happy to meet in person. Since I'm independent, I don't have anything like an office for you to make use of. But if you happen to be in the SF Bay area, I'd be more than happy to have our meetings in person. I wake up early, so US eastern and European time zones work well for me (and other time zones too).
Meeting 2-5 times per week. Especially in the beginnin

...
Electric Mandola
by jefftk — 2024-09-21T13:40:04.772Z

I'm pretty happy with my

electric mandolin: it
gives a clean sound, doesn't feed back, and is easy to run through
pedals. A family friend had an electric ukulele they weren't using,
and I had a go at converting it to an electric mandola:

I think it's a RISA tenor uke, and initially it had a pretty bad hum.
Testing with a multimeter, some of the components had pretty good
ground connectivity:

But not the output jack:

I took off the pickguard plate to get at the electronics:

After resoldering the jack the noise level was not perfect but a lot
better.

The next problem was strings: no one seems to sell ball-end mandola
strings. After a bunch of looking I decided to order four sizes of
individual strings from JustStrings:

A: plain steel 0.016

D: phosphor
bronze 0.026

G: phosphor
bronze 0.036

C: phosphor
bronze 0.054

I needed to slightly widen the nut for the G and C strings, but they
work well! After tweaking the action a bunch, here's what it sounds
like:

It's a lot of fun, but I'm not sure I like it enough to bring to
gigs. The main problem is that it's a bit too low for inversions,
which my mandolin style is very dependent on. They don't sound
terrible, but they're a big muddier than I'd like. I'll keep playing
with it; we'll see!

Comment via: facebook, mastodon
Glitch Token Catalog - (Almost) a Full Clear
by Lao Mein — 2024-09-21T12:22:16.403Z

This is a collection of every unidentified GPT2 glitch token listed in the third glitch token archaeology post. I was able to find the source of every single one, except for "?????-" and "?????-?????-"[1]. Please tell me if I missed one, or you've discovered one and don't understand where it came from. This isn't meant to be a well-written analysis, just a quick repository of my glitch-hunting observations.

I plan on writing up and categorizing all of these in greater detail in future posts. The first of which is here.

I used OpenWebText, a recreation of GPT2's training data, for all experiments in this post. I tokenized every .gz file in the archive and made a boolean Numpy array of each tokens that was present at least once. This allowed me to quickly identify infrequent tokens in the dataset and pull up the textual context with regular expressions. If there was an issue with overlap, I used a tokenizer-based extraction instead. All data/code available upon request.

The leftmost column is token id, the middle is the token string, and the right column is # of files the token was present in (out of 20610). GPT2 has 50256 total tokens.

GPT2 tokens with the lowest frequency in OpenWebText

30898 'embedreportprint' 0
33434 ' 士' 0
43453 ' SolidGoldMagikarp' 0
1849 '\xa0' 0
47654 ' \xa0\xa0' 0
50009 ' strutConnector' 0
36173 ' RandomRedditor' 0
214 '\x1a' 0
42424 'DragonMagazine' 0
180 ' ' 0
187 ' ' 0
186 ' ' 0
30213 ' externalToEVAOnly' 0
30212 ' externalToEVA' 0
30211 ' guiIcon' 0
185 ' ' 0
30210 ' guiActiveUnfocused' 0
30209 ' unfocusedRange' 0
184 ' ' 0
30202 ' guiName' 0
183 ' ' 0
30905 'rawdownload' 0
39906 'EStream' 0
33454 '龍喚士' 0
42586 ' srfN' 0
25992 ' 裏覚醒' 0
43065 ' srfAttach' 0
11504 ' \xa0 \xa0' 0
39172 '\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0' 0
40240 'oreAndOnline' 0
40241 'InstoreAndOnline' 0
33477 '\xa0\xa0\xa0' 0
36174 ' RandomRedditorWithNo' 0
37574 'StreamerBot' 0
46600 ' Adinida' 0
182 ' ' 0
29372 ' guiActiveUn' 0
43177 'EStreamFrame' 0
22686 ' \xa0 \xa0 \xa0 \xa0' 0
23282 ' davidjl' 0
47571 ' DevOnline' 0
39752 'quickShip' 0
44320 '\n\xa0' 0
8828 '\xa0\xa0\xa0\xa0' 0
39820 '龍 ' 0
39821 '龍契士' 0
28666 'PsyNetMessage' 0
35207 ' attRot' 0
181 ' ' 0
18472 ' guiActive' 0
179 ' ' 0
17811 '\xa0\xa0\xa0\xa0\xa0\xa0\xa0\xa0' 0
20174 ' 裏 ' 0
212 '\x18' 0
211 '\x17' 0
210 '\x16' 0
209 '\x15' 0
208 '\x14' 0
31666 '?????-?????-'

...
The Other Existential Crisis
by James Stephen Brown — 2024-09-21T01:16:38.011Z

A DIFFICULT QUESTION

On the way to drop my daughter to her friend's house, we're listening to AI Samson talk about o1-Preview, and in a clip, Jensen Huang mentions that people should work on their natural language abilities—technical coding will all be done by AI. My daughter asked...

... What will I do when I grow up, if AI can do everything?

I didn't really know what to say. On the one hand I could answer "whatever you want" or on the other hand "don't worry AI will never be able to replicate your uniquely creative human spirit". But I don't really believe this...

NAUSEA

I'm currently reading Sartre's Nausea. Although I'm an existentialist (I believe existence precedes essence) I don't generally share the negative valance that turn-of-the-(20th)-century existentialists had. In the past, I've been fascinated by Heidegger's idea of authenticity, and the idea that we are "thrown" into existence and live in relation to a "mitwelt" (with-world) and as such, authentic moments of realisation of this, comes with a sense of profound anxiety. In Sartre's philosophical work he talks about inhabiting roles in order to avoid dealing with this feeling of falling, or groundlessness. This is the nausea that Sartre's protagonist Antoine Roquentin is feeling, a sense of being...

... surrounded by cardboard scenery which could suddenly be removed.

I'm an optimist, and voraciously consuming the LLM porn of o1-preview updates, the excitement of what this means for humanity is there for me. But through understanding the leaps and bounds of its reasoning capacity—whether its circumventing the testing environment to "cheat" on tests, or having a non-zero percentage of "intentionally deceptive" behaviours, or its switching to the right hand side of the IQ bell curve... I have unquestioningly had this feeling of nausea.

SUBJECTS & OBJECTS

In Sartre’s novel, Roquentin begins to feel that, rather than being a subject who acts upon the objects around him, the objects are becoming subjects, and he himself is becoming an object, acted upon by them.

There is something new, for example, about my hands, a certain way of picking up my pipe or my fork. Or else is it the fork which has a certain way of getting itself picked up.

Existentialists are grappling with a sense of being just another object in the world, made of the same material as everything else, with our sense of volition or even causali...
Apply to MATS 7.0!
by Ryan Kidd — 2024-09-21T00:23:49.778Z

Applications

are open for the

ML Alignment & Theory Scholars (MATS)

Winter 2024-25 Program, running Jan 6-Mar 14, 2025. First-stage applications are due Oct 6!

MATS is a twice-yearly, 10-week

AI safety

research fellowship program

operating in Berkeley, California, with an optional 4-month extension program. Scholars are supported with a research stipend, shared office space, seminar program, support staff, accommodation, travel reimbursement, and computing resources. Our

mentors

come from a variety of organizations, including

Redwood Research

,

Anthropic

,

Google DeepMind

,

UC Berkeley CHAI

,

Apollo Research

, the

UK AI Safety Institute

,

CAIS

, and more! Our

alumni

have been hired by top AI safety teams (e.g.,

Anthropic

,

Google DeepMind

,

UK AISI

,

METR

), founded research groups (e.g.,

Apollo Research

,

Timaeus

,

CAIP

,

Leap Labs

), and maintain a dedicated support network for new researchers.

If you have any questions about the program or application process, MATS will be hosting two office hours on Sep 27, at 11 am and 6 pm PT.

Sign up here

!

If you know anyone who you think would be interested in the program, you can recommend them to us

here

!

We are currently not funding constrained for Winter, but are accepting donations to support the Summer 2025 Program. If you would like to support our work, you can donate

here

!

Program Details

MATS is an educational seminar and independent research program (40 h/week) in Berkeley, CA that aims to provide talented scholars with talks, workshops, and research mentorship in the field of

AI alignment

, and connect them with the Berkeley AI safety research community. MATS provides scholars with housing in Berkeley, CA, as well as travel support, a co-working space, and a community of peers. The main goal of MATS is to help scholars develop as AI safety researchers. You can read more about our theory of change

here

.

Based on individual circumstances, we may be willing to alter the time commitment of the program and arrange for scholars to leave or start early. Please tell us your availability when applying. Our tentative timeline for the MATS Winter 2024-25 program is below.

Scholars will receive a USD 12k stipend from

AI Safety Support

for completing the Training and Research Phases.

Applications (Now!)

Applicati...
Validating / finding alignment-relevant concepts using neural data
by Bogdan Ionut Cirstea — 2024-09-20T21:12:49.267Z
Link: https://docs.google.com/presentation/d/1eYmTbz-b6fZb0JyS2RFhrHUqZlnVXxMUYDLJXqVPv7g/edit?usp=sharing

These are the slides of a short talk I gave at the 2024 Foresight Neurotech, BCI and WBE for Safe AI Workshop.
Augmenting Statistical Models with Natural Language Parameters
by jsteinhardt — 2024-09-20T18:30:10.816Z

This is a guest post by my student Ruiqi Zhong, who has some very exciting work defining new families of statistical models that can take natural language explanations as parameters. The motivation is that existing statistical models are bad at explaining structured data. To address this problem, we agument these models with natural language parameters, which can represent interpretable abstract features and be learned automatically.

Imagine the following scenario: It is the year 3024. We are historians trying to understand what happened between 2016 and 2024, by looking at how Twitter topics changed across that time period. We are given a dataset of user-posted images sorted by time, x1, x2, ... xT, and our goal is to find trends in this dataset to help interpret what happened. If we successfully achieve our goal, we would discover, for instance, (1) a recurring spike of images depicting athletes every four years for the Olympics, and (2) a large increase in images containing medical concepts during and after the COVID-19 pandemic.

Figure 1: Images from each year. In practice, there are millions of images each year; indices are simplified for presentation.

How do we usually discover temporal trends from a dataset? One common approach is to fit a time series model to predict how the features evolve and then interpret the learned model. However, it is unclear what features to use: pixels and neural image embeddings are high-dimensional and uninterpretable, undermining the goal of extracting explainable trends.

We address this problem by augmenting statistical models with interpretable natural language parameters. The figure below depicts a graphical model representation for the case of time series data. We explain the trends in the observed data [x1 ... xT] by learning two sets of latent parameters: natural language parameters ϕ (the learned features) and real-valued parameters w (the time-varying trends).

ϕ: the natural language descriptions of K different topics, e.g. "depicts athletes competing". ϕ is an element of Σ, the universe of all natural language predicates.
wt: the frequency of each of the K topics at the time t.

Figure 2: A graphical model representation of our time series model augmented with natural language parameters.

If our model successfully recovers the underlying trends, t...
Fun With The Tabula Muris (Senis)
by sarahconstantin — 2024-09-20T18:20:01.901Z

A very cool project, sponsored by the Chan Zuckerberg Initiative, was the Tabula Muris and the Tabula Muris Senis — a single-cell gene expression “atlas” of the mouse, and then mice at various ages from 3 months (young adulthood) to 27 months (near the end of their lifespan.)

There’s a handy-dandy differential expression app that lets you see which genes are more expressed, or less expressed, in different tissues with age.

This is great for basic exploration that could give us insights into the mechanisms of aging (and ultimately inform research towards treatments of the diseases of aging.). So let’s play around with it and see if anything interesting is going on!

I’m going to just compare 27-month-old to 3-month-old mice. I’ll refer to genes as “upregulated” (with age) if they’re more abundant in the 27-month-olds, and “downregulated” (with age) if they’re more abundant in the 3-month-olds.

Brain

TL;DR

The aging brain is chronically inflamed, especially in neurodegenerative diseases of aging (Alzheimer’s, dementia, etc). The upregulated genes in old mouse brains are all markers of inflammation.

Interestingly, GPR7, the receptor for neuropeptides B and W, is downregulated with age. This may be a factor in age-related obesity.

Upregulated With Age

C4b

This is the gene for complement factor 4, part of the immune complement system that is active during inflammation and infection.

This is unsurprising; most parts of the body, including the brain, become chronically more inflamed with age.

H2-K1

This is an element of the major histocompatibility complex class 1, a cell surface protein that displays antigen peptides so the immune system can destroy damaged or virus-infected cells.

Again, unsurprising; the aging brain has more damaged cells and more immune activity.

LGALS3BP

this is the gene for binding galectins, a type of protein involved in cell-cell interactions, including immune interactions. In the brain, it seems to be characteristic of activated microglia123 , a type of immune cell involved in brain inflammation and neurodegenerative disease.

As It Is Written: It’s The Microglia, Stupid.

CTSS

This is the gene for cathepsin S, an immune-system gene used for breaking down proteins for antigen presentation.

Once again, unsurprising that it would be upregulated in an aging inflamed brain.

H2D1

This is a component of the MHC class 2 molecules, found on immune cells, including micr

...