trentmkelly/LessWrong-43k huggingface dataset:
FLI podcast series, "Imagine A World", about aspirational futures with AGI
by Jackson Wagner — 2023-10-13T16:07:39.001Z

Last year, the Future of Life Institute organized the "AI Worldbuilding Contest", where they challenged competitors to envision a future where humanity has safely navigated the development of artificial general intelligence, avoiding major wars and catastrophes. The winning entries were imaginative and diverse; I profiled a few of my favorite entries here.

Now, FLI is following up on the project with "Imagine A World"! This new podcast series digs deeper into the eight winning entries, exploring the different ideas and solutions that each team used to build their distinct, aspirational futures.

Here's a little bit about each of the first four episodes; I'll cover the next four once they all come out!

"What if new governance mechanisms helped us coordinate?"

(Apple Podcasts, Soundcloud, Spotify, and here is the original worldbuilding entry)

This is the first episode of FLI's series, and it's also my own team's entry! Here are some of my favorite moments from the podcast interview:

At the 11:15 mark, we introduce the overall historical arc of our scenario, then move on to talk about how new institutions like prediction markets and charter cities could improve humanity's ability to make wise decisions and coordinate on global problems.
At 27:12, talking about how a rapidly-changing landscape of AI capabilities will create a lot of uncertainty about who the ultimate "winners" vs "losers" will be. This uncertainty might actually make international coordination easier, creating a situation like the "veil of ignorance" thought experiment from philosopher John Rawls.
Around 38:30, Holly and I gush over some of sci-fi stories that we took inspiration from -- books like Ada Palmer's Too Like The Lightning, Ursula LeGuin's The Dispossessed, and Yudkowsky's Inadequate Equilibria, that take improvements in "social technology" just as seriously as advancements in science and engineering.

"What if narrow AI fractured our shared reality?"

(Apple Podcasts, Soundcloud, Spotify, and here is the original worldbuilding entry)
This world, from Michael Vassar, brings a strong dose of Robin-Hanson-esque pessimism about human motivations and realpolitik about the structure of societies and institutions. In this world, cutting-edge AGI hasn't yet been rolled out to the world, instead being cautiously tested and tweaked for alignment. But mea...
To open-source or to not open-source, that is (an oversimplification of) the question.
by Justin Bullock — 2023-10-13T15:10:21.429Z

*** This an edited and expanded version of

a post I made on X

in response to

GovAI’s new report“Open-Sourcing Highly Capable Foundation Models”

I think the report points in the right direction, but also leaves me with some additional questions. Also, thanks for significant feedback from @David_Kristoffersson, @Elliot_Mckernon, @Corin Katzke, and @cwdicarlo ***

From my vantage point the debate around open-sourcing foundation models became heated as Yann LeCun began advocating for open-sourcing (in particular) Meta's foundation models. This prompted a knee-jerk reaction in the AI Safety community.

The arguments went something like "of course open-sourcing foundation models is a good idea, just LOOK at all the BENEFITS open-sourcing has given us!" for the "pro" crowd, and something like "of course open-sourcing foundation models is a terrible idea, just THINK about how it increases RISK" for the "anti" crowd.

Given this, I was excited to see the release of

GovAI’s new report

which, as Elizabeth A. Seger highlights in

their brief summary

, outlines both the noted benefits and risks of open-sourcing more generally, and how these benefits and risks might be applied in particular to foundation models. In this report titled “Open-Sourcing Highly Capable Foundation Models” Seger, along with her numerous co-authors walk us through these benefits and risks and also explore alternative policies that arguably provide similar benefits as open-sourcing while mitigating the risks of open-sourcing foundation models.

After reading that report, I have a few summarizing thoughts:

To open source or not to open source foundation models is a false dichotomy. Instead, there is a gradient of options to consider.

This is something that should be more obvious but does seem to be lost in the current debate. The gradient runs from fully closed to fully open and includes additional categories including gradual/staged release, hosted access, cloud-based/API access, and downloadable. (“Box 1: Further research is needed to define open-source gradients” from the paper illustrates this well.) It’s worth noting that even

Meta’s Llama2 is not fully open

.

Structured access seems to be a particularly useful option. It provides many of the benefits of fully open-sourcing while protecting against some of the risks of both fully

...
Combination Lock Boxes
by jefftk — 2023-10-13T12:50:12.874Z

Traditionally we control access to our houses with "something you
have": a key. This works pretty well, until you have a visitor arrive
when you're not home to give them a key. The traditional way to
handle this is to hide a key (under the mat, in a flower pot, etc),
turning this into "something you know" (the information about where
the key is hidden).

I've talked to a few people recently who've been enthusiastic about
swapping traditional keyed house locks out for combination locks, and
while I do see the appeal they're somewhat expensive and require
batteries. But they're definitely a more secure way to implement
access conditional on "something you know" than hiding a key.

Another option to consider, however, is a combination key lock box,
sometimes called a "realtor box":

You set a combination on it, and then store a spare key inside. It is
more hassle than a pure combination lock for access, but for a our
house it works well. People generally carry their keys, but we still
have a reasonably secure option if someone forgets theirs or a guest
arrives while we're out.
Circle of Support (Oct 14th @ 10am PST)
by Alexei — 2023-10-13T09:24:41.068Z

Having spent some extended time in this community back in 2010s, I am well familiar with a lot of the stressors we can experience:

Feeling of doom from AI (and other x-risks)
Pressure to be more "effective" and more "altruistic"
Fear of (or anger at) aging and dying
Intra-community drama
Burnout
Unusual mental states

With these in mind, I would like to offer my self as a listener and a supporter for you.

Where: EA GatherTown:

https://app.gather.town/app/Yhi4XYj0zFNWuUNv/EA%20coworking%20and%20lounge

Find me (Alexei Andreev) on the right side of the screen in the list of users and click "Follow".

When: Saturday Oct 14th from 10am to 11am PST.
How can the world handle the HAMAS situation? 
by Annapurna — 2023-10-13T09:15:40.654Z

We are a few hours away from a likely ground incursion / invasion of Israeli Defence Forces (IDF) into Northern Gaza. For the past week I've been trying to figure out a more reasonable way that the world can deal with Hamas, as an alternative to what the Israeli government is considering doing. I am having trouble coming up with a rational option aside from status quo (leave the group as is) or Israel's option (eliminate no matter the cost).

What other option exists to deal with a group with the following fact sheet (and please correct anything that is factually wrong in the statements below):

Hamas was elected democratically in 2006 and stayed in power since then.

The leaders of Hamas and their families live in relative luxury while the average Gazan lives near the poverty line.

They promote radicalized education in Gaza, half of which are under the age of 19.

They use humanitarian aid for war (for example, digging up water pipes and turning them into missiles).

They strategically place their military operations near school, hospitals, residences, and commercial areas.

They convince Gazans to stay in urban areas and not to head to places IDF is advising them to go to in order not to get bombed.

They recruit minors to their cause and send them in paragliders to murder people (and be murdered by IDF)

They have the sympathy of many, EVEN IN ISRAEL, because of the civilian death toll every time there is an IDF operation.

I really don't know what can be done that is better than either option presented above.
UVic AI Ethics Conference
by TristanTrim — 2023-10-13T07:31:23.370Z

Hello! I'm part of the AI club at the University of Victoria. We want to help expose students and professors at our university to the ideas of AI alignment and AI ethics, and so we are putting together an online conference this November 25th.

Our club website

Eventbrite page

We are looking for presenters, so if you, or anyone you know, may be interested in presenting on an AI Ethics topic, please reach out : ) You can comment here or pm me, or use the contact page on our website.

I'm most interested in preventing X-Risk from not having sufficient alignment in agents capable of recursive self improvement to superintelligence, but I feel it is also important to host discussion on AI related ethics concerns that are already affecting peoples lives. Moreover, if you have another topic you feel would make sense at our conference, I'd be happy to hear about it. We'd love talks from people working in alignment labs, academia, or with independent funding, to inspire those of us interested in working on these issues, and to help our peers realize this can be a viable, even profitable, career.

Also, of course, if you just want to attend our conference, I invite you to do so!

Thanks!
LW UI features you might not have tried
by Elizabeth — 2023-10-13T03:04:57.542Z

Every once in a while I see a comment describing a problem with lesswrong that is fixable with existing features, if only the user knew about them. This post is an attempt to make these highly useful features more widely known.

I am technically a mod on lesswrong, which is why I know about these features. But in practice my only modding is giving opinions to the core team and recommending a few things a year for curation. I have absolutely no power over features. If you would like to criticize or praise these features, please direct that towards the core team.

Personalize your front page algorithm

Seeing way too much AI? Care about postmortem posts even when they have 5 karma? Never want to hear about decision theory again? You can make this happen by going to the front page -> click “customize feed” -> add a tag -> select a bias to apply to that tag. You can hide all, halve their karma for purposes of your front page, +25 their karma for purposes of your front page, or make up a custom modifier.

These don't affect the post, author, or anyone else's feed, just what you see.

Ban users from your posts

If you have 1000 karma, you can ban individuals from specific personal posts (as in, posts that haven’t been set to the front page). At 2000 karma, you have the same options for front page posts.

EDIT 10/13: To enable this feature, set moderation guidelines first.

To do this, click on the three dot menu icon in the upper left hand corner of a comment. If you have the power, this will contain options to ban a user from this post or all posts.

Bookmark page (thanks Jozdien)

The Bookmarks page that contains all of your bookmarks, and your reading history and your vote history.
Revisiting Guide Dogs and Blindness Prevention
by jefftk — 2023-10-13T02:30:07.155Z

In early effective altruism it

was common

for
people

to
give

the
example of deciding between donating to fund the training of guide
dogs in wealthy countries or to reduce blindness in very poor
countries. For example, here's what Giving What We Can

used
to say:

For example, suppose we want to help those who are blind. We can help
blind people in a developed country like the United States by paying
to train a guide dog. This is more expensive than most people realize
and costs around $50,000 to train a dog and teach its recipient how to
make best use of it. In contrast, there are millions of people in
developing countries who remain blind for lack of a cheap and safe eye
operation. For the same amount of money as training a single guide
dog, we could completely cure enough people of Trachoma-induced
blindness to prevent a total of 2,600 years of blindness.

That writing went up sometime in 2011
or earlier, and I think it's where this comparison entered proto-EA, but if you know of earlier
usage I'd be curious!

Over time, however, EAs have mostly moved away from this
comparison. Instead global poverty EAs are more
likely to give comparisons like:

A typical US income vs how much the world's poorest people live
on.

How much the US is willing to pay to save the life of one of
its citizens vs how much it costs to save a life where that's
cheapest.

What the UK National Health Service is willing to spend per
Quality-Adjusted Life Year (QALY) vs what it costs to provide a
similar benefit to someone in a very poor region.

What these have in common is that there are charities working in these
areas with strong evidence to support their cost-effectiveness. With
GiveDirectly you can send money to people who are living on under
$1/day, or with the Against Malaria Foundation you can provide
bednets that cut
malaria deaths in ~half, at around $5k per life saved
(~$100/QALY).

On the other hand, while GiveWell and other EA groups have
looked into vision
surgery, there are no high quality evaluations. [1] What do you do if you say something like "while a guide dog
costs $X you can restore someone's sight for $Y << $X" and
someone asks where they can donate to make that happen? Much better
to give actionable comparisons.

[1] You might think of Hellen Keller International, another GiveWell
recommendation, but looking...
Paper: Understanding and Controlling a Maze-Solving Policy Network
by TurnTrout — 2023-10-13T01:38:09.147Z
Link: https://arxiv.org/abs/2310.08043

Mrinank, Austin, and Alex wrote a paper on the results from Understanding and controlling a maze-solving policy network, Maze-solving agents: Add a top-right vector, make the agent go to the top-right, and Behavioural statistics for a maze-solving agent.

Abstract: To understand the goals and goal representations of AI systems, we carefully study a pretrained reinforcement learning policy that solves mazes by navigating to a range of target squares. We find this network pursues multiple context-dependent goals, and we further identify circuits within the network that correspond to one of these goals. In particular, we identified eleven channels that track the location of the goal. By modifying these channels, either with hand-designed interventions or by combining forward passes, we can partially control the policy. We show that this network contains redundant, distributed, and retargetable goal representations, shedding light on the nature of goal-direction in trained policy networks.

We ran a few new experiments, including a quantitative analysis of our retargetability intervention. We'll walk through those new results now.

Retargeting the mouse to a square involves increasing the probability that the mouse goes to the target location. Therefore, to see how likely the mouse is to visit any given square, Alex created a heatmap visualization:

Normalized path probability heatmap. The normalized path probability is the geometric average probability, under a policy, along the shortest path to a given point. It roughly measures how likely a policy is to visit that part of the maze.
The color of each maze square shows the normalized path probability for the path from the starting position in the maze to the square. In this image, we show the "base probabilities" under the unmodified policy.

For each maze square, we can try different retargeting interventions, and then plot the new normalized path probability towards that square:

Notice the path from the bottom-left (where the mouse always starts) to the top-right corner. This is the top-right path. Looking at these heatmaps, it's harder to get the mouse to go farther from the top-right path. Quantitative analysis bears out this intuition:

Overall, these new results quantify how well we can control the policy via the internal goal representations which we identified.

Thanks to Lisa Thiergart for helping handle funding...
OPTIC: Announcing Intercollegiate Forecasting Tournaments in SF, DC, Boston
by Saul Munn — 2023-10-13T01:36:48.331Z

TLDR: We’re excited to announce that OPTIC is running three intercollegiate forecasting competitions this fall — in the Bay Area (Nov 4), DC (Nov 18) and Boston (Dec 2). Register here!

What does a tournament look like?

Think 1-day hackathon/olympiad/debate tournament, but for forecasting the future — teams predict on topics ranging from geopolitics to celebrity twitter patterns to financial asset prices, and the best forecasters get thousands of dollars in cash prizes and exclusive internships at Metaculus.

Day of, teams give probabilistic predictions for a couple hours on about 30 given questions (with breaks for lunch & speakers). Teams are 3-5 competitors, and we’ll place you on a team if you don’t already have one in mind. A few months after the tournament, all questions resolve and winners are announced/awarded: until then, you can track how your team is doing in real time.

Forecasting, cash prizes & exclusive internships, opportunities to meet other intellectually minded students, more — register here!

Competition Logistics

We’ll be running competitions in the Bay Area (Nov 4), DC (Nov 18), and Boston (Dec 2).

Day of, teams give probabilistic predictions for a couple hours on about 30 given questions (with breaks for lunch & speakers). Teams are 3-5 competitors, and we’ll place you on a team if you don’t already have one in mind. A few months after the tournament, all questions resolve and winners are announced/awarded: until then, you can track how your team is doing in real time.

To get a taste of what a competition looks like, you can:

Look over our previously used questions
Watch Seth Blumberg’s speech on the future of forecasting (Seth is a behavioral analyst at Google and the head of Google’s internal prediction market, speaking in a personal capacity)
If you want a deep look into the nuts and bolts, check out our postmortem of the pilot competition!

Sponsorship & Funding

This semester, OPTIC received funding from Open Philanthropy to run forecasting tournaments. We’re also sponsored by the forecasting platform Metaculus and the quant firm SIG. We’re grateful for their support and excited that participants & sponsors get to interface with with each other through OPTIC.

Previously, we’ve received funding from the Long-Term Future Fund and from the ACX Forecasting Mini-Grants (facilitated through the Manifund Impact Market).

About OPTIC

OPTIC b...
Progress links digest, 2023-10-12: Dyson sphere thermodynamics and a cure for cavities
by jasoncrawford — 2023-10-13T00:41:40.024Z
Link: https://rootsofprogress.org/links-digest-2023-10-12

I’ve been traveling for a while, so this is a long one, covering the last ~month. I tried to cut it down, but there have been so many amazing announcements, opportunities, etc.! Feel free to skim and jump around:

From the Roots of Progress fellows

Connor O’Brien and Adam Ozimek have a new proposal, the Chipmaker’s Visa, “to ensure the fabs we’re spinning up have the specialized talent they need immediately” (via @cojobrien)
In Defence of a High-Energy Future (via @mboudry)
Anna-Sofia Lesiv on desalination (via @annasofialesiv)
Does the Vegas Sphere Mark the End of Monument Stagnation?

On the Progress Forum

“The Knowledge Machine delves deep into the interplay between systems thinking and human nature, starting with a thought-provoking idea: most people lack the inherent drive to pursue knowledge.” Science Despite the Fragility of Scientists
“There exists a fundamental mismatch in scale that means no one really represents the interests of the overall Bay Area, nor has the power to govern it coherently. This is causing a fundamental breakdown.” It’s Time for Greater San Francisco

Events

DC, Oct 16: Anticipating the Future with Virginia Postrel and Jim Pethokoukis
San Francisco, Dec 1–3: I’ll be at the Foresight Institute’s Vision Weekend USA, speaking on an “Existential Hope” panel

Prizes

A $10,000 challenge for writings of a positive future made with biology. Deadline extended to Oct 15 (via @NikoMcCarty, @HomeworldBio)
For science journalists: The Good Science Project is joining with Johns Hopkins to sponsor a set of awards for reporting on science policy issues (via @stuartbuck1)
The Progress Prize. “The best blog gets £5,000 plus other goodies” (via @Tom_Westgarth15)
“Will give £100 to anyone who can come up with a catchy alternative term for ‘the British Industrial Revolution’ that I can use without cringing” (@antonhowes)

Opportunities

“Are you a people-wrangling project manager who wants to bring more awesome science and technology into the world? Then check out our new short term job posting here” (via @Spec__Tech)
Ideas Matter: An 8-week writing fellowship for biology (via @NikoMcCarty)
“We’re looking for someone to help us at @fiftyyears with a research project on the history of technological progress, its future, and the potential contribution to a thriving future” (via @sethbannon)
“I’m working on charting technological progress in different mediums (electron

...
What do Marginal Grants at EAIF Look Like? Funding Priorities and Grantmaking Thresholds at the EA Infrastructure Fund
by Linch — 2023-10-12T21:40:17.654Z

The EA Infrastructure Fund (EAIF) offers targeted grants to amplify projects based on effective altruism principles, enhancing their access to talent, capital, and knowledge. They are currently fundraising for the next 6 months' grant distribution. This article provides insight into how EAIF prioritizes projects to fund. I describe hypothetical grants that EAIF might fund based on how much they raise.

For donors who believe that the current marginal EAIF grants are better than marginal funding of all other organizations, please consider

donating

to EAIF! [1]This is an especially good time to donate, as donations are matched 2:1 by Open Philanthropy (OP donates $2 for every $1 you donate).

Other updates

As

mentioned in August

, EA Funds and Open Phil are distancing themselves from each other.

To help with the transition, Open Phil is switching from a model where they directly give grants to EA Funds (including the EA Infrastructure Fund) to one where they provide 2:1 matching.

Relatedly, Max Daniel (who works at Open Phil) has

stepped down

as chair of EAIF.
We’ve written a (rather overdue) payout report for EAIF and aim to publish it before the end of this month.
We’re

fundraising

for EAIF, as it is rather

funding constrained

. Donations will be matched 2:1 by Open Phil until the end of January.
We plan to make the strategy of EAIF more conceptually crisp going forwards, likely with a specific focus towards working on “principles-first EA.” More details forthcoming.
We’re likely to try to hire a chair for EAIF soon, probably after we conclude a hiring round for the

full-time LTFF chair role

. We have not yet decided whether it makes sense for the EAIF chair to be full-time.

Intro

This article follows a similar

post about funding priorities at the Long-Term Future Fund

(LTFF), and similar methodologies and caveats apply here.

In particular:

This article is primarily aimed at donors, not grantees. Compatibility between an applicant and their proposed project, including personal interest and enthusiasm, plays a crucial role in the project’s success. Therefore, we discourage tailoring your applications to match the higher tiers of this list.
My own view is that the conceptual grounding for EAIF grants is a bit worse than that for LTFF grants. Thus, the reasoning behind previous EAIF grants might be a bit harder to generalize than for LTFF

...