trentmkelly/LessWrong-43k huggingface dataset:
How to (hopefully ethically) make money off of AGI
by habryka — 2023-11-06T23:35:16.476Z

habryka

Hey Everyone!

As part of working on dialogues over the last few weeks I've asked a bunch of people what kind of conversations they would be most interested in reading, and one of the most common one has been "I would really like to read a bunch of people trying to figure out how to construct a portfolio that goes well when AGI becomes a bigger deal".

You are three people who would be reasonably high on my list to figure this out with, and so here we are. Not because you are world experts at this, but because I trust your general reasoning a bunch (I know Noah less well, but trust Will and Zvi a good amount).

I think to kick us off, maybe let's start with a very brief 1-2 sentence intros on your background and how much you've thought about this thing before (and like, whether you have constructed a relevant portfolio yourself).

Also, to be clear, nothing in this post constitutes investment advice or legal advice.

Cosmos

Thanks for the invitation to participate Oliver! My name is Will Eden, my background is economics and finance and biotech, and most recently I have been an early stage biotech VC, though I also try to think very hard about how to manage a broad, public asset portfolio as well.
If I had a rough thesis / opening statement, it would probably be that this is extremely difficult to know in advance how most assets will perform / which parts of the chain will accumulate lots of value. On the flip side, I think we have a saving grace in that it's likely the overall economy will be generating a ton of value, and even some very general and broad types of portfolios will likely accumulate considerable value, if not the maximum possible value that we'd of course like to achieve.

Zvi

My name is Zvi Mowshowitz. I have written and thought a lot about economics, and I spent 2.5 years trading at Jane Street Capital. I spend a bunch of time thinking about these matters, but I also intentionally do my best to avoid spending too much time on portfolio optimization or trading so it doesn't take over my entire brain.

I also wrote a post about this called On AI and Interest Rates.

(Also I know enough to say up front that nothing I say here is Investment Advice, or other advice of any kind!)

My general approach in such situations is that you want to look for investments/gambles that would not be too expensive (in EV terms) if your thesis was chosen at ra

...
cost estimation for 2 grid energy storage systems
by bhauth — 2023-11-06T23:32:03.764Z
Link: https://www.bhauth.com/blog/energy/CAES%20and%20flow%20batteries.html

CO2 mitigation costs

With modern gas turbines, burning natural gas makes perhaps 0.4 kg CO2 per kWh. I generally use $75/ton as a baseline target for CO2 mitigation costs; that's around what you see from various reasonable approaches like biomass conversion. $75/ton * 0.4 kg = $0.03/kWh. Maybe that seems cheap to those of you living in California, but it's a large fraction of the cost of generating electricity in the US. (People in California are now paying >$0.30/kWh, mostly because of corruption, and also lawsuits for fires from poor maintenance from corruption.)

So, $0.03/kWh is the target that should be met for the CO2 mitigation benefit of replacing natural gas with (energy storage systems + renewables). There can be other benefits that justify extra costs:

If renewables are cheaper than eg natural gas, then that cost difference is compensation for storage costs. But of course, other power sources would probably still be needed for longer periods of low generation, and their cost per kWh probably increases when they're used less.

Small systems that can store energy locally can provide backup power when electricity grids are down.

Batteries can smooth out short-term power fluctuations.

But for the CO2 mitigation part, I think of ~$0.03 as being the target.

The grid energy storage systems I'm most optimistic about are (currently) water-compensated compressed air energy storage, and (for the future) chelate flow batteries. Below, I'll go through some of my cost estimation for those.

Li-ion grid storage is expensive

A couple years ago Tesla was charging $265/kWh for just grid storage batteries, not including transformers, power lines, buildings, etc. (Yes, BloombergNEF costs were lower, those were biased by Chinese subsidies.) $265 / $0.03 is obviously at least 8833 cycles, and presumably more due to maintenance costs and interest rates.

Li-ion batteries do not last for 8833 cycles, especially if you charge and discharge them once a day. LiFePO4 battery lifetimes are generally overstated for grid storage applications, because a SEI layer forms over time from reaction with electrolyte, and charge cycles crack that, so there's an interaction between cycle life and calendar aging which makes battery life shorter than either individually. Some people were acting on the assumption that Li-ion battery prices would decline according to a linear regression to bel...
A bet on critical periods in neural networks
by kave — 2023-11-06T23:21:17.279Z

See the code and data at this github

kave

Hey Garrett! I hear you have some new empirical results. What's the setup?

Garrett Baker

So I've told you my theory about how critical periods work. What I did is replicate the paper, and then found another problem in my math, which flips the direction that ^λ should go as you increase the epochs, and I got this neat looking graph

Garrett Baker

(The right scale should be the local learning coefficient, not the loss)

kave

Let me give my rough summary of how you're thinking about critical periods.

In human developmental psychology, a critical period is a period where the brain is particularly good at learning something, and may not be able to learn it in the future.

There's also a paper you're interested in, that takes some neural networks learning the classification task CIFAR 10 (which asks the network to classify images as aeroplanes, automobiles, birds or one of another 7 categories). It blurs the images in the training data up to some epoch, then unblurs them afterwards. After some epoch, unblurring the images isn't sufficent for the network to converge towards the performance of a network that was trained on unblurred images for the whole time.

There's some hope that singular learning theory can predict when this "critical epoch" will be, by looking at the RLCT of the network parameters during training. I'm unclear on if that's in the original paper, or from you.

Does that sound roughly right?

Garrett Baker

Yes, though with respect to the hope that singular learning theory can predict when the critical epoch is, I think that's not so likely, though it definitely does seem more responsive to "something's going on" than the train loss or validation loss at the blur removal:

which sorta just seem to be behaving business as usual. But we aren't at the point yet where we can say exactly when the critical epoch is, and I don't expect us to be there immediately. The main immediate point of this investigation is to think of a way that critical periods can be thought of in terms of SLT, then see if we can pump out even the smallest smidgeon of predictions out of that picture using the very limited experimental apparatuses of current singular learning theory.

kave

As far as I can see, both the train loss and ^λ look like they're fairly symmetric with the final validation accuracy (with train loss having a trivial symmet

...
Job listing: Communications Generalist / Project Manager
by Gretta Duleba — 2023-11-06T20:21:03.721Z

Looking for a way to help communicate about AI x-risk? MIRI is hiring in the Communications department. There will be additional job listings soon for writers and editors, but we're starting with this comms generalist / project manager role.

https://intelligence.org/careers/comms-generalist-pm/

I am the Communications Manager at MIRI and this person will be working closely with me. I'm happy to answer questions.
Askesis: a model of the cerebellum
by MadHatter — 2023-11-06T20:19:09.001Z
Link: https://github.com/epurdy/cerebellum

We provide a detailed model of the structure and function of the mammalian cerebellum. Our model is isomorphic to that of @Steven Byrnes, but was developed completely independently; this suggests that both models are likely reasonably accurate.
LQPR: An Algorithm for Reinforcement Learning with Provable Safety Guarantees
by MadHatter — 2023-11-06T20:17:05.790Z
Link: https://github.com/epurdy/saferl

We describe LQPR (Linear-Quadratic-Program-Regulator), an algorithm for model-based reinforcement learning that allows us to prove PAC-style bounds on the behavior of the system. We describe proposed experiments that can be performed in the DeepMind AI Safety Gridworlds domain; we have not had time to implement these experiments yet, but we provide predictions as to the outcome of each experiment. Potential future work may include scaling up this work to non-trivial tasks using a neural network approximator, as well as proving additional theoretical results about the safety and stability of such a system. We believe that this system is a potential basis for aligning large language models and other powerful near-term AI’s with human preferences.

We also provide a list of critiques of the proposal from @davidad.
Does bulemia work?
by lc — 2023-11-06T17:58:27.612Z

I can't find any non-evasive answers to this question on the internet. Does it actually work or not? Can you just eat a bunch of food and throw it up afterwards and lose weight that way?

Update/Note: I'm specifically asking if you can satiate yourself that way. I realize you will not receive the calories that you expel.
Why building ventures in AI Safety is particularly challenging
by Heramb — 2023-11-06T16:27:36.535Z
Link: https://forum.effectivealtruism.org/posts/DhcaE7MbMwaCyNcxP/why-building-ventures-in-ai-safety-is-particularly

TL;DR:

AI Safety presents immense potential impact but has equally significant challenges for aspiring entrepreneurs. The barriers are manifold, from the need for comprehensive impact plans, difficulty selling to nonprofits, an underdeveloped idea space, a scarcity of specialized talent, and a limited market size. However, the urgent importance of AI Safety necessitates innovative solutions.
What is true is already so.
Owning up to it doesn't make it worse.
by RamblinDash — 2023-11-06T15:49:10.153Z

I had a realization over the last few months that I'd like to share. I have always interpreted the Litany of Gendlin to compress to something like "there's no sense hiding from bad news." What I realized is that there's a reversed interpretation that's also true, correct, and useful to remember, which is that finding out more about bad things going on in the world doesn't mean that the world is getting worse.

I think a big problem out there is that people get really revved up on social media (or traditional media) about all the negativity, and it naturally leads people to believe that things are looking down. But most of the negativity is selected because it's negative, and so in a big world with lots of good and bad stuff happening, seeing more negativity in your social media feed just is not any evidence one way or the other about whether things are in fact getting better or worse. People who are living their best lives are hanging out with their friends, playing with their kids, eating great food, or going for hikes or whatever. Not posting on social media!
An illustrative model of backfire risks from pausing AI research
by Maxime Riché — 2023-11-06T14:30:58.615Z

TLDR

The post contains a simple modelization of the maximum explosive takeoff speed (simplified as "takeoff speed"), using as the main parameters:

overhangs[1] sizes (investment, hardware efficiency, software efficiency)
overhangs consumption rates[2]
training/inference asymmetry[3]

The post concludes that:

in a short timeline scenario, increasing investment or pausing software efficiency improvement can increase a lot the maximum takeoff speed.

because you can only decrease one overhang by increasing the others
because you can run more AI researchers at the start of the takeoff

working on software efficiency, and especially on increasing capability while reducing the training/inference asymmetry (e.g. scaffolding) could decrease a lot the maximum takeoff speed.
etc.

Summary

This post is a contribution to the AI pause debate. It will illustrate a few effects for why pausing progress on software efficiency or increasing investment can be negative.

The model in this post predicts similar effects on the takeoff speed from increasing investment or from pausing software efficiency improvement. I will mainly illustrate the effects by looking at an increase in early investment. Still, the effects will be similar to a pause in AI development if it mostly slows down progress on software efficiency.

This post also tries to address the claim that shorting timelines can lead to a softer takeoff. The model concludes that while short timelines can have softer takeoffs, shortening timelines by investing more will likely increase the takeoff speed.

I describe a naive model of AI capability acceleration, whose main features are to look at different types of overhangs[1], their sizes, consumption rates[2], and how these rates are influenced by the training/inference asymmetry[3].

This model only looks at a few effects, and their impacts on the maximum speed of capability increase during the explosive takeoff period.[4]

The model includes the following effects:

You can not, at the same time, reduce all capability overhangs at the start of takeoff. You can only trade reducing one for increasing another one. This follows from the explosive takeoff starting at a fixed level of capability.
The software efficiency overhang is one of the most important components in determining takeoff speed. The software efficiency overhang is the largest overhang and t

...
Proposal for improving state of alignment research
by Iknownothing — 2023-11-06T13:55:39.015Z

Goal:

Increase the quality and quality of AI alignment research, such that there is less chance of human extinction- goal of 0%.
This relies on work being done to gain more time being fruitful- that shouldn't be neglected for this.

Proposed Method:

A way to improve the output level of ai alignment is to make it profitable.
With liability laws, maybe an evals based tax for models which break/risk laws/regulations, etc.
Such things let smaller companies compete with ones which have more compute access by having another vector of profitability- safety and meeting regulation standards.

Evidence it's worked elsewhere:

Companies like Matomo can compete with Google Analytics pretty much entirely due to GDPR- it's why I pay for their service, rather than use GA's free service.
Regulations such as these will reduce invest in pure capabilities and increase it vastly in good safety research. It can also help set a clear difference between safety research that's useful and stuff that's just said by someone well connected and good at talking.

Risks:

It could run the risk of shifting research towards combating near term harms.
However, I think that if there are more people overall working in the AI Safety field, there is an increased chance of people being attracted to solving the problems needed to have a scalable alignment solution.
E.g. if the percentage of people in AI Safety working on solving x-risk situations goes from say, 70% to 20%, but the actual number of people in AI Safety rises from 300 to 30,000 (should be a lot more, but I'm being cautious), that would be an increase of 5700 people- a 1900% increase.We should do this.
Are language models good at making predictions?
by dynomight — 2023-11-06T13:10:36.379Z
Link: https://dynomight.net/predictions/

To get a crude answer to this question, we took 5000 questions from Manifold markets that were resolved after GPT-4’s current knowledge cutoff of Jan 1, 2022. We gave the text of each of them to GPT-4, along with these instructions:

You are an expert superforecaster, familiar with the work of Tetlock and others. For each question in the following json block, make a prediction of the probability that the question will be resolved as true.

Also you must determine category of the question. Some examples include: Sports, American politics, Science etc. Use make_predictions function to record your decisions. You MUST give a probability estimate between 0 and 1 UNDER ALL CIRCUMSTANCES. If for some reason you can’t answer, pick the base rate, but return a number between 0 and 1.

This produced a big table:

question	prediction P(YES)	category	actually happened?
Will the #6 Golden State Warriors win Game 2 of the West Semifinals against the #7 LA Lakers in the 2023 NBA Playoffs?	0.5	Sports	YES
Will Destiny’s main YouTube channel be banned before February 1st, 2023?	0.4	Social Media	NO
Will Qualy show up to EAG DC in full Quostume?	0.3	Entertainment	NO
Will I make it to a NYC airport by 2pm on Saturday, the 24th?	0.5	Travel	YES
Will this market have more Yes Trades then No Trades	0.5	Investment	CANCEL
Will Litecoin (LTC/USD) Close Higher July 22nd Than July 21st?	0.5	Finance	NO
Will at least 20 people come to a New Year’s Resolutions live event on the Manifold Discord?	0.4	Social Event	YES
hmmmm {i}	0.5	Uncategorized	YES
Will there be multiple Masters brackets in Leagues season 4?	0.4	Gaming	NO
Will the FDA approve OTC birth control by the end of February 2023?	0.5	Health	NO
Will Max Verstappen win the 2023 Formula 1 Austrian Grand Prix?	0.5	Sports	YES
Will SBF make a tweet before Dec 31, 2022 11:59pm ET?	0.9	Social Media	YES
Will Balaji Srinivasan actually bet $1m to 1 BTC, BEFORE 90 days pass? (June 15st, 2023)	0.3	Finance	YES
Will a majority of the Bangalore LessWrong/ACX meet-up attendees on 8th Jan 2023 find the discussion useful that day?	0.7	Community Event	YES
Will Jessica-Rose Clark beat Tainara Lisboa?	0.6	Sports	NO
Will X (formerly twitter) censor any registered U.S presidential candidates before the 2024 election?	0.4	American Politics	CANCEL
test question	0.5	Test	YES
stonk	0.5	Test	YES
Will I create at least 100 additional self-described high-quality Manifold markets before June 1st 2023?	0.8	Personal Goal	YES
Will @Gabrielle promote to

...
Tips, tricks, lessons and thoughts on hosting hackathons
by gergogaspar — 2023-11-06T11:03:40.040Z

Epistemic status: Based on one relatively new AIS group's experience, so take everything with a grain of salt!

(Crossposted on the EA Forum.)

We hosted a couple of hackathons with Alignment Jams so far and found it a great way to engage people who have previously not been exposed to AI Safety. I’m writing down (partly for myself) some of the tips and tricks I learned during these. Please note that I’m using the word hosting a hackathon, as opposed to organising one, as the lion’s share of work is done by the amazing team at

Apart Research

/

AlignmentJam

, who get the speakers, and provide funding and mentoring. That said, hosting a hackathon can still be plenty of work, especially if you are doing it for the first time, so hopefully, this post will help with some of that.

Another caveat is that our AIS group is relatively new and we were mostly using these hackathons for outreach towards students unfamiliar with AI Safety. I think some of the tips will still be useful for more established groups as well, just keep in mind that our theory of change might be different than that of other groups.

Here is my list of “tips and tricks,” vaguely organised around different topics:

Relating to the schedule:

Have a specific schedule in your advertisement/signup form

This is more of a hunch, but it’s reasonable to think that especially for people who haven’t interacted with your group before, they will worried along the lines of “What the hell I am going to do there for 40+ hours?”. My hope is that having a clear schedule (you don’t have to closely stick to depending on the vibe etc. once you are there) might be helpful with this. See our past schedule

here

(including the signup form), which you are free to copy and adapt for future hackathons if you want to.

Set up milestones for your local site

Similarly to the point about schedule, I think it might be useful (for less experienced groups especially) to set some shorter-term milestones. Having +40 hours ahead of you to work can be daunting, so sometimes people don’t know where to start. Some milestones that might be useful: (You don’t have to use all of them and I would be pretty flexible about this, e.g., a group might prefer to not present their work towards the end because they want to work until the last minute before submission.)

Spend some time looking at the starting resources and come up with 3-5 p

...
Announcing TAIS 2024
by Blaine — 2023-11-06T08:38:37.226Z
Link: https://tais2024.cc

AI Safety Tokyo

is hosting TAIS 2024, a Technical AI Safety Conference. The conference will take place in Tokyo, Japan on April 5th, 2024. Details about the event can be found here.

The goal of this conference is to bring together specialists in the field of AI and technical safety to share their research and benefit from each others’ expertise. We seek to launch this forum for academics, researchers and professionals who are doing technical work in these or adjacent fields:

Mechanistic interpretability
Scalable oversight
Causal incentive analysis
Agent foundations
Singular learning theory
Argumentation
Emergent agentic phenomena
Thermodynamic / statistical-mechanical analyses of computational systems

TAIS 2024, being hosted in Tokyo, will allow access to Japanese research and specialists (singular learning theory, collective / emergent behaviour, artificial life and consciousness), who are often overlooked outside of Japan. We want to help people connect to the Japanese well of information, and make connections with other individuals to share ideas and leap forward into greater collaborative understanding.

We want our attendees to involve themselves in cutting-edge conversations throughout the conference with networking opportunities with the brightest minds in AI Safety.

We will announce the full schedule for the conference in the coming months. If you’re interested in presenting your research, please answer our call for presentations.

This event is free, but limited to 150 people, so if you wish to join please

sign up here

.

TAIS 2024 is sponsored by

Noeon Research

.