trentmkelly/LessWrong-43k huggingface dataset:
CAIDP Statement on Lethal Autonomous Weapons Systems
by Heramb — 2024-11-30T18:16:53.183Z
Link: https://forum.effectivealtruism.org/out?url=https%3A%2F%2Fwww.linkedin.com%2Fposts%2Fcenter-for-ai-and-digital-policy_caidp-lethal-autonomious-weapons-systems-activity-7262308184843763712-Dxgr%3Futm_source%3Dshare%26utm_medium%3Dmember_desktop%20

Summary:

Key Concerns with Lethal Autonomous Weapons

⚠️ ❗ Unpredictability and Lack of Control

⚠️ ❗ Exponential Lethality

⚠️ ❗ Ethical and Legal Implications

Recommendations

1️⃣ Immediate Moratorium: Enact a temporary ban on deploying LAWS until
comprehensive regulations are established.

2️⃣ Classification as WMDs: Classify lethal autonomous weapons, like 'loitering' AI missile systems, under weapons of mass destruction due to their scalable lethality.

3️⃣ Ban Non-Compliant AI Systems: Prohibit AI systems that cannot adhere to
international human rights and humanitarian laws.

4️⃣ Monitoring Framework: Implement standardized reporting and allow independent oversight of AI in military operations.

5️⃣ Appoint a UN Special Rapporteur on AI and Human Rights: Encourage
transparency and human rights alignment

6️⃣ Promote Democratic Accountability: Ensure civilian control and prevent
unverified AI systems from influencing military decisions.

"The majority of UN Member States support regulating LAWS despite opposition from a few powerful countries. Immediate action is crucial to prevent an AI arms race, protect human rights, and maintain international peace and security."
Launching Applications for the Global AI Safety Fellowship 2025!
by Aditya_SK — 2024-11-30T14:02:16.537Z

TLDR;

Applications

are accepted until December 31, 2024 on a rolling basis, to join a 3-6 month, fully-funded research program in AI safety. Fellows work with some of the world’s leading AI safety labs and research institutions. This will be a full-time, in-person placement, after which there may be opportunities to continue the engagement full time based on mutual fit and fellow's performance.

Learn more and apply to be a fellow

here

, or refer someone you think would be awesome for this

here

. We’re also looking for Talent Identification Advisors (or Consultants) – find out more about the role

here

.
—

Impact Academy

is an organisation focused on running cutting-edge fellowships to enable global talent to use their careers to contribute to the safe and beneficial development of AI.

Impact Academy’s Global AI Safety Fellowship is a 3-6 month fully-funded research program for exceptional STEM talent worldwide.

1⃣ Fellows will work with leading AI safety organisations to advance the safe and beneficial development of AI. Some of our placement partners are the Center for Human Compatible AI (CHAI), FAR.AI, Conjecture, and the UK AISI.

Applications are being accepted on a rolling basis until 31 December 2024, but early applications are strongly encouraged. The exact start date of the Fellowship will be decided by the candidate and the placement organization.

Fellows will work in person with partner organisations, subject to visa. If fellows experience visa delays, we will enable them to work from our shared offices at global AI safety hubs.

Ideal candidates for the program will have-

Demonstrated programming proficiency (e.g. >1 year of relevant professional experience).
A strong background in ML (e.g. full-semester university courses, significant research projects, or publications in ML).
A track record of excellence (e.g. outstanding achievements in academics or other areas).
An interest in pursuing research to reduce the risks from advanced AI systems.

Please apply even if you do not meet all qualifications! Competitive candidates may excel in some areas while developing in others.

Fellows will receive a comprehensive financial package to cover their salary, living expenses and research costs, along with dedicated resources for building foundational knowledge in AI safety, regular mentorship, and 1:1 coaching calls with the Impact Academy t...
Exporting Facebook Comments, Again
by jefftk — 2024-11-30T12:40:07.339Z

I want comments on my social media crossposts to show up on my blog as
a comment section, and mostly this works well: modern systems (

Mastodon,

Bluesky,

LessWrong,
etc) provide APIs where you can load the replies associated with a
post. On the other hand, older systems like Facebook are more locked
down: they want to keep your content inside the platform as part of
their

economic
moat.

Still, Facebook will show all the comments on a post to anyone who
visits it, even if logged out. You have to dismiss a popup and click
"show more" and "see replies" a few times, but it's all public. At
times I've written
scripts to
export the comments, but they're quite brittle: Facebook doesn't
design their pages to be easy to scrape, and so my code has relied on
incidental things that only happen to work.

Even though this is not a permanent solution, I've had another go at
writing a comment exporter (code).
It's not as thorough as past times: I couldn't figure out easy was to
get the timestamp or links to the comment on Facebook, and I've left
both out. I also had to switch my opt-out from working on user id to
user name, which is less robust. But it works! I've gone back
through June 2019, fetching comments for any posts where I was missing
them.
Mathematical Futurology: From Pseudoscience to Rigorous Framework
by Wenitte Apiou — 2024-11-30T03:27:26.152Z

Futurology has long been dismissed as a pseudoscience, occupying the same intellectual space as astrology in many academic circles. This skepticism isn't entirely unwarranted - much of futurism consists of unfalsifiable predictions, trend extrapolation without theoretical foundation, and what Philip Tetlock would call "vague verbiage."

My interest in formalizing futurology isn't purely theoretical. While at Vanderbilt University's School of Engineering, I received approval for a self-designed concentration in Mathematical Futurology. The fact that an engineering school was open to such an unconventional individual study path suggested there might be merit in developing more rigorous approaches to future studies. (Though in a perhaps fitting twist for a futurist, I ended up dropping out - the irony of abandoning a traditional path while studying how institutions adapt to change isn't lost on me.)

The history of simulation-based forecasting provides interesting precedent for adding rigor to future studies. From Jay Forrester's system dynamics to modern climate models, we've seen how mathematical modeling can capture complex system behavior and generate actionable insights about possible futures. The key distinction? These approaches rely on rigorous formalization of underlying mechanisms rather than pure extrapolation.

I believe we're at an inflection point in forecasting capabilities. Modern AI developments, particularly in areas like neural ODEs, differentiable programming, and multi-agent simulation, are about to supercharge our ability to model complex systems. We're moving from simple parameter-tuned simulations to learning-based models that can discover their own parameters and even underlying mechanisms from data.

This potential for enhanced simulation capabilities led me to question: Could we develop a more rigorous foundation for futurology? Last year, I collaborated with ChatGPT to write a textbook titled "Mathematical Futurology" (an exercise in AI-augmented research itself). The goal was to build from first principles, starting with philosophical foundations:

Epistemology: How can we know anything about future states?

Ontology: What is the nature of future possibilities?

Logic: How do we reason about systems that contain reasoners?

Just as Newton formalized mechanics with his laws of motion, I thought it would be interesting to propose fundamental la...
(The) Lightcone is nothing without its people: LW + Lighthaven's big fundraiser
by habryka — 2024-11-30T02:55:16.077Z

Update Jan 19th 2025: The Fundraiser is over! We had raised over $2.1M when the fundraiser closed, and have a few more irons in the fire that I expect will get us another $100k-$200k. This is short of our $3M goal, which I think means we will have some difficulties in the coming year, but is over our $2M goal which if we hadn't met it probably meant we would stop existing or have to make very extensive cuts. Thank you so much to everyone who contributed, seeing so many people give so much has been very heartening.

TLDR: LessWrong + Lighthaven need about $3M for the next 12 months. Donate or send me an email, DM, signal message (+1 510 944 3235), or public comment on this post, if you want to support what we do. We are a registered 501(c)3, have big plans for the next year, and due to a shifting funding landscape need support from a broader community more than in any previous year. [1]

I've been running LessWrong/Lightcone Infrastructure for the last 7 years. During that time we have grown into the primary infrastructure provider for the rationality and AI safety communities. "Infrastructure" is a big fuzzy word, but in our case, it concretely means:

We build and run LessWrong.com and the AI Alignment Forum.[2]
We built and run Lighthaven (lighthaven.space), a ~30,000 sq. ft. campus in downtown Berkeley where we host conferences, research scholars, and various programs dedicated to making humanity's future go better.
We are active leaders of the rationality and AI safety communities, running conferences and retreats, participating in discussion on various community issues, noticing and trying to fix bad incentives, building grantmaking infrastructure, talking to people who want to get involved, and lots of other things.

In general, Lightcone considers itself responsible for the end-to-end effectiveness of the extended rationality and AI safety community. If there is some kind of coordination failure, or part of the engine of impact that is missing, I aim for Lightcone to be an organization that can jump in and fix that, whatever it is.

Doing that requires a non-trivial amount of financial capital. For the next 12 months, we expect to spend around $3M, and in subsequent years around $2M (though we have lots of opportunities to scale up if we can get more funding for it). We currently have around $200k in the bank.[3]

Lightcone is, as far as...
Sexual Selection as a Mesa-Optimizer
by Lorec — 2024-11-29T23:34:45.739Z

[ Note 1: This post is a follow-up to "Thoughts on Evo-Bio Math and Mesa-Optimization: Maybe We Need To Think Harder About 'Relative' Fitness?" ]

[ Note 2: Warning: the writing style of this post is kind of jank. It blends formal and informal register, uses punctuation loosely and italics liberally, and contains long sections of extended quotes. If that kind of thing tends to put you off, please don't read this post. Unless you're in a mood for risk. Or unless you're so into evolutionary biology, or mesa-optimizer theory, that you'd read any new hypothesis no matter how poorly written. ]

[ Note 3: The earlier parts of this post are somewhat an entry into my series of woefully incomplete histories.

Since biology hypotheses - though when fully understood they vary wildly in their explanatory power! - are unusually hard to crisply distinguish from one another [given the high dimensionality of their study-objects], correct attribution of priority is even harder in biology than it is in pure game theory. Please inform me of any unambiguous inaccuracies; otherwise, I issue a repeat, intensified apology for the blurriness here, along with my duplicated conviction that LW having a version of the history part is worth the fog. ]

[ Note 4: I can imagine, if I try, the publication of this post being bad for capabilities. But given that all the [vague outlines of] techniques discussed are, as I understand it, already well-known to ML, if it is bad for capabilities it will probably be bad by way of emotionally inspiring somebody. So: if you find yourself emotionally inspired by this post to work on AI capabilities, try this one. If that's not to your taste, and you're still feeling inspired to go work on capabilities - because, for example, you see my argument about the capabilities part of the mesa-optimizer but you're not sure the inner and outer optimizer I describe in this post are that misaligned with each other, not really, not if you just - . . . then I wish you happiness, but I can only wish you failure. You have missed the entire point, and I publish this only in spite of you. Your posts will not age well. ]

"This subject of sexual selection has been treated at full length in the present work, simply because an opportunity was here first afforded me. I have been struck with the likeness of many of the half-favourable criticisms on sexual selection, with those whic

...
INTELLECT-1 Release: The First Globally Trained 10B Parameter Model
by Matrice Jacobine — 2024-11-29T23:05:00.108Z
Link: https://www.primeintellect.ai/blog/intellect-1-release

We're excited to release INTELLECT-1, the first 10B parameter language model collaboratively trained across the globe. This represents a 10× scale-up from our previous research and demonstrates that large-scale model training is no longer confined to large corporations but can be achieved through distributed, community-driven approaches. The next step is scaling this even further to frontier model sizes and ultimately open source AGI.
You should consider applying to PhDs (soon!)
by bilalchughtai — 2024-11-29T20:33:12.462Z

TLDR: In this post, I argue that if you are a junior AI safety researcher, you should consider applying to PhD programs in ML soon, especially if you have recently participated in an

AI

safety

upskilling

or

research

program

like MATS or ARENA and might be interested in working on AI safety long term, but don't have immediate career plans. It is relatively cheap to apply and provides good future option value. I don’t argue that you should necessarily do a PhD, but some other

posts

do. I also point out that starting a PhD does not lock you in if better opportunities arise. PhD application deadlines for Fall 2025 start are coming up soon; many application deadlines are December 15th, though some are as early as next week. For the uninitiated, I provide a step by step guide to applying at the end of this post.

Applying to PhD programs might, in expectation, be worth your time.

This might be true even if you are not sure you:

Want to do a PhD.
Think being a researcher is the best way for you to contribute to AI safety.
Think you even want to work on AI safety.

This is provided that you assign sufficient probability to all of the above, and you don't currently have better plans. PhD applications can be annoying, but in general

applying

is

cheap

, and can often teach you things. A PhD might be something you might want to start in a year. Applying now gives you the optionality to do this, which future you might thank you for.

What is the basic case for a PhD?

A PhD is the world's default program for training researchers.

"Making AI go well" is a hard problem, which we do not know how to solve. It therefore requires research. A PhD lets you work on, and get empirical feedback on, your own research ideas, which seems important for building "research taste": the important skill of developing and evaluating research ideas.
More ambitiously, AI safety lacks "

research leads

" -- people who are capable of coming up with and leading new research agendas; such people often have PhDs (though note this is not the only or even most common path for PhD graduates).

Being a PhD student might put you in a better environment to do productive research.

Compared to being an independent researcher; you have access to more resources: an advisor (who will either be somewhere between very helpful, not very helpful, and net negati

...
Understanding Emergence in Large Language Models
by  — 2024-11-29T19:42:43.790Z

Recent research into large language models (LLMs) has revealed fascinating patterns in how these systems develop capabilities. While initial discussions of "emergent abilities" suggested sudden, discontinuous jumps in performance, closer analysis reveals a more nuanced picture that warrants careful examination.

The Data Behind Emergence

The concept of emergence in LLMs was first systematically studied through the BIG-bench benchmark. Initial observations suggested that capabilities like emoji movie interpretation appeared to emerge suddenly at certain model scales. For instance, between 10^10 and 10^11 parameters, models showed dramatic improvements in their ability to interpret emoji sequences representing movies. [1]

However, these apparent discontinuities deserve closer scrutiny. When we examine the actual data:

The choice of evaluation metric significantly impacts whether abilities appear emergent. When using exact string matching, capabilities seem to appear suddenly. However, when using multiple-choice evaluations or examining log likelihoods of correct answers, we see much more gradual improvements.
Looking at aggregate performance across benchmarks (as seen in GPT-3's development), the improvement curves are actually smooth rather than discontinuous.

Understanding Multi-Step Reasoning

One compelling explanation for apparently emergent behavior comes from examining multi-step reasoning. Consider a task requiring ten consecutive correct reasoning steps. Even if a model's ability to perform individual reasoning steps improves smoothly, the probability of completing the entire chain successfully can show a sharp, seemingly discontinuous jump.

This matches what we observe in practice. Tasks requiring multiple steps of reasoning or complex chains of thought tend to show more apparent "emergence" than simpler tasks, even though the underlying capabilities may be improving gradually.

Scaling Laws and Practical Limitations

Recent research from Google DeepMind (the Chinchilla paper) has shown that optimal training requires about 20 tokens of training data for each parameter in the model. This creates practical limits on scaling:

A 100-trillion parameter model would require approximately 2,000 trillion tokens of training data
This would need about 180 petabytes of high-quality text
For comparison, the entire Common Crawl dataset is only about 12 petabytes

These constrai...
The 'Road Not Taken' in the Multiverse
by Jonah Wilberg — 2024-11-29T19:01:51.775Z

Summary: The Many-Worlds interpretation of quantum mechanics can help us feel better about choosing between mutually exclusive options.

Previously I’ve suggested that when we think about the ethical implications of the

many-worlds interpretation (MWI)

of quantum mechanics, the kinds of implications we should expect are ones about how to feel in certain ethical situations, and what kinds of character traits or ‘virtues’ we should try to develop.

In this post I’m going to focus in on the implications it has for how we feel about mutually exclusive life choices.

The Road Not Taken

The fact that the human condition contains mutually exclusive options is often seen as cause for regret and anxiety. Robert Frost’s poem

The Road Not Taken

is a classic expression of this.

The narrator is “sorry I could not travel both” of the two roads that diverge in a yellow wood, and expects to look back on his choice “with a sigh”.

In addition to being mutually exclusive, the roads are also hard to see down due to the fact that they “bent in the undergrowth”, and are “really about the same” in terms of their attractiveness.

This reflects another aspect of the choices that cause such feelings. It is because both choices seem more or less equally good, given our limited knowledge, that being forced to choose one or the other provokes such anxiety.

The poem strikes a chord because it expresses something we all feel at times. Life does contain such choices.

Often such choices are relatively trivial. Choosing a dessert from a menu, for instance. Choosing a film to watch on a night out. Choosing a holiday destination.

But sometimes they can be significant. Choosing an undergraduate degree. Choosing who to vote for. Choosing a hobby. Choosing between two life partners.

Sometimes too the choice may be whether to take some particular course of action or not to do so. Getting married, for example, or having children.

Travelling both roads

Does MWI allow us to travel both roads, and thereby feel differently about such decisions?

A sceptic might

argue

that “Decision and decoherence are entirely orthogonal concepts.”

As Eliezer puts it:

You don’t split when you come to a decision in particular, any more than you particularly split when you take a breath. You’re just splitting all the time as the result of decoherence, which has nothing to do with choices….It’s

...
The Big Nonprofits Post
by Zvi — 2024-11-29T16:10:06.938Z

There are lots of great charitable giving opportunities out there right now.

The first time that I served as a recommender in the Survival and Flourishing Fund (SFF) was back in 2021. I wrote in detail about my experiences then. At the time, I did not see many great opportunities, and was able to give out as much money as I found good places to do so.

How the world has changed in three years.

I recently had the opportunity to be an SFF recommender for the second time. This time I found an embarrassment of riches. Application quality was consistently higher, there were more than twice as many applications, and essentially all applicant organizations were looking to scale their operations and spending.

That means the focus of this post is different. In 2021, my primary goal was to share my perspective on the process and encourage future SFF applications. Sharing information on organizations was a secondary goal.

This time, my primary focus is on the organizations. Many people do not know good places to put donations. In particular, they do not know how to use donations to help AI go better and in particular to guard against AI existential risk. Until doing SFF this round, I did not have any great places to point them towards.

(Not all the applications were about AI. There is also a lot of attention to biological existential and catastrophic risks, some to nuclear threats, and a number of applications that took entirely different approaches.)

Table of Contents

Organizations where I have the highest confidence in straightforward modest donations now, if your goals and model of the world align with theirs, are in bold.

A Word of Warning.

Use Your Personal Theory of Impact.

Use Your Local Knowledge.

Unconditional Grants to Worthy Individuals Are Great.

Do Not Think Only On the Margin, and Also Use Decision Theory.

And the Nominees Are.

Balsa Research.

Don’t Worry About the Vase.

Organizations Focusing On AI Non-Technical Research and Education.

The AI Futures Project.

Lightcone Infrastructure.

Effective Institutions Project (EIP).

Artificial Intelligence Policy Institute (AIPI).

Psychosecurity Ethics at EURAIO.

Pallisade Research.

AI Safety Info (Robert Miles).

Intelligence Rising.

Convergence Analysis.

Longview Philanthropy.

Organizations Focusing Primary On AI Policy and Diplomacy.

Cente

...