trentmkelly/LessWrong-43k huggingface dataset:
Refinement of Active Inference agency ontology
by Roman Leventov — 2023-12-15T09:31:21.514Z
Link: https://arxiv.org/abs/2312.07547

[Submitted on 6 Dec 2023]

Active Inference and Intentional Behaviour

Karl J. Friston, Tommaso Salvatori, Takuya Isomura, Alexander Tschantz, Alex Kiefer, Tim Verbelen, Magnus Koudahl, Aswin Paul, Thomas Parr, Adeel Razi, Brett Kagan, Christopher L. Buckley, Maxwell J. D. Ramstead

Abstract:

Recent advances in theoretical biology suggest that basal cognition and sentient behaviour are emergent properties of in vitro cell cultures and neuronal networks, respectively. Such neuronal networks spontaneously learn structured behaviours in the absence of reward or reinforcement. In this paper, we characterise this kind of self-organisation through the lens of the free energy principle, i.e., as self-evidencing. We do this by first discussing the definitions of reactive and sentient behaviour in the setting of active inference, which describes the behaviour of agents that model the consequences of their actions. We then introduce a formal account of intentional behaviour, that describes agents as driven by a preferred endpoint or goal in latent state-spaces. We then investigate these forms of (reactive, sentient, and intentional) behaviour using simulations. First, we simulate the aforementioned in vitro experiments, in which neuronal cultures spontaneously learn to play Pong, by implementing nested, free energy minimising processes. The simulations are then used to deconstruct the ensuing predictive behaviour, leading to the distinction between merely reactive, sentient, and intentional behaviour, with the latter formalised in terms of inductive planning. This distinction is further studied using simple machine learning benchmarks (navigation in a grid world and the Tower of Hanoi problem), that show how quickly and efficiently adaptive behaviour emerges under an inductive form of active inference.

From the introduction:

Specifically, this paper differentiates between three kinds of behaviour: reactive, sentient, and intentional. The first two have formulations that have been extensively studied in the literature, under the frameworks of model-free reinforcement learning (RL) and active inference, respectively. In model-free RL, the system selects actions using either a lookup table (Q-learning), or a neural network (deep Q-learning). In standard active inference, the action selection depends on the expected free energy of policies (Equation 2), where the expectation is ov

...
EU policymakers reach an agreement on the AI Act
by tlevin — 2023-12-15T06:02:44.668Z

On December 8, EU policymakers

announced an agreement on the AI Act

. This post aims to briefly explain the context and implications for the governance of global catastrophic risks from advanced AI. My portfolio on Open Philanthropy’s AI Governance and Policy Team includes EU matters (among other jurisdictions), but I am not an expert on EU policy or politics and could be getting some things in this post wrong, so please feel free to correct it or add more context or opinions in the comments!

If you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via

this short Google form

.

Context

The AI Act was first introduced in April 2021, and for the last ~8 months, it has been in the “trilogue” stage. The EU Commission, which is roughly analogous to the executive branch (White House or 10 Downing Street), drafted the bill; then, the European Parliament (sort of like the U.S. House of Representatives, with seats assigned to each country by a population-based formula) and the Council of the EU (sort of like the pre-17th-Amendment U.S. Senate, with each country's government getting one vote in a complicated voting system)[1] each submitted proposed revisions; then, representatives from each body negotiated to land on a final version (analogous to

conference committees

in the US Congress).

In my understanding, AI policy folks who are worried about catastrophic risk were hoping that the Act would include regulations on all sufficiently capable GPAI (general-purpose AI) systems, with no exemptions for open-source models (at least for the most important regulations from a safety perspective), and ideally additional restrictions on “very capable foundation models” (those above a certain compute threshold), an idea floated by some negotiators in October. In terms of the substance of the hoped-for regulations, my sense is that the main hope was that the legislation would give the newly-formed AI Office substantial leeway to require things like

threat assessments/dangerous capabilities evaluations

and

cybersecurity measures

, with a lot of the details to be figured out later by that Office and by standard-setting bodies like

CEN-CENELEC’s JTC-21

.

GPAI regulations

appeared in danger of being excluded

after Mistral,...
Where Does Adversarial Pressure Come From?
by quetzal_rainbow — 2023-12-14T22:31:25.384Z

You could see on LW a dialogue like this:

AI Pessimist: Look, you should apply security mindset when thinking about aligning superintelligence. If you have superintelligence, all your safety measures are under enormous optimization pressure, which will seek how to crack them.

Skeptic: I agree that if we have misaligned superintelligence, it will game whatever specification we give it and kill us or worse. But why would we have misaligned superintelligence in the first place? Where does adversarial pressure come from?

And this is a legitimate question! I always feel the statement "optimization pressure causes weird things" to be intuitive, but many people don't share that attitude and it's not as crisp as I've wanted.

Let's begin with the question: what does it mean for your security assumption to be broken? It means that your security assumption happens to be false. Security assumptions are not locks or doors; you can't break them through physical effort. If all your security assumptions are true (including "this set of assumptions is sufficient for safety"), you are safe. You are only in danger if your security assumptions are false.

If so, why do we have to deal with pressure on these assumptions? Because you are running systems whose very first task is to learn what is true and what is false.

That's it. We don't need anything else to justify necessity to treat overall situation as adversarily-optimized. It's an implementation-agnostic statement. It doesn't matter whether your system is agentic or non-agentic, symbolic or connectionist - if you increase the ability of your system to figure out things as high as possible, your system is going to shoot you in the leg if your security assumptions are false.

Old models of adversarial pressure presented something like this: the system has a plan-generator and plan-evaluator. If your evaluator is not maximally robust, your generator is going to find a plan which satisfies the evaluator but is actually bad. This construction seems unnatural: why would we want to build a generator-of-all-possible-plans? Why wouldn't we want to build a generator-of-similar-to-past-plans, at least?

But we can reframe the situation in other settings. Let's take the alignment of a DL system. Our security assumptions can be, for example:

Our training data contains only Good Things.
Our learning algorithm can learn only Good Things.
Learned Good T

...
Epoch wise critical periods, and singular learning theory
by Garrett Baker — 2023-12-14T20:55:32.508Z

This was the application I used to get into the upcoming developmental interpretability MATS stream. It is essentially a more polished version of this dialogue

Abstract

I investigated epoch-wise critical periods in neural networks from a singular learning theory lends. Hypothesizing that the cause has to do with a monotonic increasing relationship between epochs and temperature leading to periods at which, interpreting SGD as a kind of MCMC sampler, poor mixing occurs, resulting in SGD’s stationary distribution occupying a region of parameter space that doesn’t include the best parameter. This theory was experimentally supported by watching the RLCT increase only during the critical epochs, and showing we can increase when the critical period occurs by decreasing the batch size, a quantity with a far more straightforward relationship to the sampling temperature.

Introduction

I have argued in both a recent post, and dialogue that singular learning theory applied to reinforcement learning and neuroscience could serve as a bridge between the two, and allow for formally verifiable alignment between AI and human interests. In order to accomplish this goal, we need to actually know how to apply singular learning theory to developmental psychology and neuroscience.

Many animals that perform significant within-lifetime learning undergo critical periods, and so too do neural networks when trained on the appropriate tasks with the appropriate settings. If we hope to expand the theory of singular learning theory to developmental psychology and neuroscience, the study of in-silico critical periods therefore provides a well-defined and likely simple sub-problem with which to study.

Not only are the existence of critical periods impactful for the theory of neuroscience, but they're also impactful for asking about how the theory of singular learning generalizes to non-iid distributions. Critical periods arise due to distribution shifts in the training data, a major component of the modern language model fine-tuning paradigm. Those in the paper I build on here also occur epoch-wise, similarly making progress on the question of how to factor epochs into the framework of SLT.

The paper Critical Learning Periods in Deep Networks (Achille et al., 2019) is able to mimic early-childhood/infant cataract-caused permanent “amblyopia” (a weakening in the vision of an eye) by training image ...
Applications for EA Global are still open!
by Eli_Nathan — 2023-12-14T19:10:55.595Z

Applications for EA Global: Bay Area 2024 (Global Catastrophic Risks) are still open and close on January 21, at 11:59 pm PT (apply here)!

EA Global conferences last around three days and involve talks, workshops, discussions, and many chances to connect and network with people involved in effective altruism. This upcoming conference in the Bay Area will be our first one exclusively focused on global catastrophic risks (GCRs). We'll be welcoming up to 1000 attendees at the Oakland Marriott City Center and platforming high-quality content related to GCRs, including AI safety, biorisks, nuclear security, and more.

We have limited travel funding available. More information can be found on the event page and EA Global FAQ. If you have any questions, please email us at hello@eaglobal.org!
Personal Development System: Winning Repeatedly and Growing Effectively With The BIG4
by Paul Rohde — 2023-12-14T18:49:06.199Z
Link: https://blog.paul-rohde.com/personal-development-system-winning-repeatedly-and-growing-effectively-with-the-big4/

“Goals are for people who care about winning once. Systems are for people who care about winning repeatedly.”

—James Clear

James Clear eloquently highlights the power of systems, primarily focusing on habits. However, this perspective, while valuable, doesn't fully capture the expansive potential of systematic thinking in personal and professional growth. This first article in the BIRR series aims to broaden this view, presenting a more comprehensive approach to leveraging system thinking for our betterment.

Personal Preface

Like many, I experiment with various models, concepts, and methods — from popular self-help books to courses and coaching. Yet, despite these efforts, I felt stagnated, as if each method was a puzzle piece that didn't quite fit. This prompted a period of introspection, leading to a pivotal realization: the need to reimagine my approach to personal development.

I discovered the significance of viewing personal growth through a probabilistic, experimental, and systematic lens. The magic wasn't in a single intervention but in a blend of strategies, each contributing incrementally to overall growth. Every habit, action, tool, and thought forms part of an interconnected web that shapes our lives. This shift in mindset was key in transitioning from sporadic improvements to consistent, long-term development.

This insight was instrumental in creating the Personal Development System (PDS) — a perspective that acknowledges the difficulty and demandingness of effective personal development. The PDS is a scaffolding system designed to be problem- and goal-agnostic. It provides the robust and adaptable infrastructure needed to effectively navigate the ever-changing landscape of personal and professional goals and challenges. This new perspective not only enhanced my own journey but also deeply resonated with my coachees. They often express gratitude for this approach, highlighting how it has brought them clarity and measurable progress in both their personal and professional endeavors.

Combating harmful and ineffective models of personal development

If you’re reading this, it's likely you share an interest in personal development. Perhaps you've tried various methods, hopefully with some success. However, a common pitfall I've observed involves skepticism or disillusionment with personal development. People often recount their experiences, saying...
Introducing The ‘From Big Ideas To Real-World Results’: A Series for Effective Personal Development
by Paul Rohde — 2023-12-14T18:49:06.186Z
Link: https://blog.paul-rohde.com/introducing-from-big-ideas-to-real-world-progress/

Various posts cover paths to peak personal development (e.g.,

replacing guilt series

,

replacing fear series

,

multiagent models of mind series

, or

CFAR Handbook

). Each offers valuable insights and strategies. However, a significant challenge I've encountered in my coaching practice is the effective translation of these big ideas and ambitious intentions into tangible, real-world results.

Throughout my coaching experience, I've identified four common hindrances that frequently prevent effective personal development:

Confused mastery: A tendency to engage superficially with numerous concepts, leading to a lack of depth or proficiency in any. This results in a scattered approach instead of focused mastery.
Comfortable delusion: The misconception that understanding a concept equates to the ability to execute it effectively, neglecting the necessity for rigorous, effortful practice required for true mastery.
Fragmented advice: The difficulty of sifting through a myriad of disparate and often incompatible pieces of advice, complicating the creation of a cohesive, effective personal development strategy.
Poorly-tested implementation: A disconnection between engaging educational content and its practical application, underscoring the importance of field-tested guidance that works in varied real-life situations.

Each hindrance is a formidable barrier to realizing our full potential. To address these challenges, I've developed the 'From Big Ideas to Real-World Results' (BIRR) series. Leveraging nearly a

decade of coaching experience

, the BIRR series is designed to bridge the gap between ambitious aspirations and palpable progress. It provides a systematic and field-tested approach, combining effective tools with insightful analysis, to help you navigate your ever-changing landscape of personal and professional goals and challenges.

The Four Hindrances Explained

From Confused Mastery To Cultivating Discernment

The first problem is confused mastery. In an era brimming with advice from endless sources—books, online courses, podcasts, and well-intentioned peers—it becomes increasingly difficult to discern truly valuable insights from the noise. It is difficult to identify which ideas are worth mastering without becoming distracted by new shiny, cool ones. Bruce Lee famously stated, "I fear not the man who has practiced 10,000 kicks once, but I f...
Talking With People Who Speak to Congressional Staffers about AI risk
by Eneasz — 2023-12-14T17:55:50.606Z
Link: https://www.thebayesianconspiracy.com/2023/12/202-the-center-for-ai-policy-talks-government-regulation/

A conversation with Jason Green-Lowe and Jakub Kraus of the Center for AI Policy. They've met with 50+ congressional staffers in DC about AI regulation efforts and are in the process of drafting a model bill. This is a somewhat entry-level conversation, good for getting an idea of what's going on over there without getting very wonky.
Bayesian Injustice
by Kevin Dorst — 2023-12-14T15:44:08.664Z
Link: https://kevindorst.substack.com/p/bayesian-injustice

(Co-written with Bernhard Salow)

TLDR:

Differential legibility

is a pervasive, persistent, and individually-rational source of unfair treatment. Either it’s a purely-structural injustice, or it’s a type of "zetetic injustice"—one requiring changes to our practices of inquiry.

Finally, graduate admissions are done. Exciting. Exhausting. And suspicious.

Yet again, applicants from prestigious, well-known universities—the “Presties”, as you call them—were admitted at a much higher rate than others.

But you’re convinced that—at least controlling for standardized-test scores and writing samples—prestige is a sham: it’s largely money and legacies that determine who gets into prestigious schools; and such schools train their students no better.

Suppose you’re right.

Does that settle it? Is the best explanation for the Prestie admissions-advantage that your department has a pure prejudice toward fancy institutions?

No. There’s a pervasive, problematic, but individually rational type of bias that is likely at play. Economists call it “statistical discrimination” (or "screening discrimination").

But it’s about uncertainty, not statistics. We’ll call it Bayesian injustice.

A simplified case

Start with a simple, abstract example. Two buckets, A and B, contain 10 coins each. The coins are weighted: each has either a ⅔ or a ⅓ chance of landing heads when tossed. Their weights were determined at random, independently of the bucket—so you expect the two buckets to have the same proportions of each type of coin.

You have to pick one coin to bet will land heads on a future toss.

To make your decision, you’re allowed to flip each coin from Bucket A once, and each coin from Bucket B twice. Here are the outcomes:

Which coin are you going to bet on? One of the ones (in blue) that landed heads twice, of course! These are the coins that you should be most confident are weighted toward heads, since it’s less likely that two heads in a row was a fluke that that one was.

Although the proportions of coins that are biased toward heads is the same in the two buckets, it’s easier to identify a coin from Bucket B that has good chance to land heads. As we might say: the coins from Bucket B are more legible than those from Bucket A, since you have more information about them.

This generalizes. Suppose there are 100 coins in each bucket, you can choose 10 to bet on landing heads, and you ar...
AI #42: The Wrong Answer
by Zvi — 2023-12-14T14:50:05.086Z

With the year ending and my first Vox post coming out, this week was a natural time to take stock. I wrote my first best-of post in a long time and laid out my plans for my 501c(3).

It was also another eventful week. We got a lot more clarity on the OpenAI situation, although no key new developments on the ground. The EU AI Act negotiators reached a compromise, which I have not yet had the opportunity to analyze properly. We got a bunch of new toys to play with, including NotebookLM and Grok, and the Gemini API.

I made a deliberate decision not to tackle the EU AI Act here. Coverage has been terrible at telling us what is in the bill. I want to wait until we can know what is in it, whether or not that means I need to read the whole damn thing myself. Which, again, please do not force me to do that if there is any other way. Somebody help me.

Table of Contents

I have a post in Vox about Biden’s executive order and the debates surrounding it. I found out from this process that traditional media can move a lot slower than I am used to, so this is not as timely as I would have liked. They also help you to improve your work. So it is not as timely as I would like, but I am happy with the final product.

This week also includes OpenAI: Leaks Confirm the Story. We get more color on what happened at OpenAI, and confirmation of many key facts. The picture is clear.

Also this week but not about AI, Balsa Update and General Thank You regarding my other policy efforts, and The Best of Don’t Worry About the Vase since I hadn’t done that in six years.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. The search for the good life continues.

Language Models Don’t Offer Mundane Utility. What we have is a failure to grok.

GPT-4 Real This Time. Lazy on winter break, looking around for… newspapers?

The Other Gemini. API it out, and nothing is ever a coincidence.

Fun With Image Generation. Don’t train on me.

Deepfaketown and Botpocalypse Soon. AI robocalls, boyfriends, honest deepfakes.

They Took Our Jobs. Journalists and teachers. Where are you providing value?

Get Involved. Fellowships, an invitation to talk, an excellent potential hire.

Introducing. Claude for Sheets.

In Other AI News. Union deals, ByteDance claims, robots trained zero shot.

Quiet Speculations. Predictions are hard, perhaps you need a cult.

...
Some for-profit AI alignment org ideas
by Eric Ho — 2023-12-14T14:23:20.654Z

Summary

This is a brain dump of some for-profit AI alignment organization ideas, along with context for why I believe a for-profit alignment organization can make a big contribution to AI safety. This is far from a complete list, and I welcome ideas and feedback. Also, if anyone wants to or is working on any of these ideas, I’d be happy to support in any way I can!

Context

I'm Eric, formerly co-founder of RippleMatch, an AI recruiting company with ~$80M raised, millions of users, and ~10% of the Fortune 500 as customers. I made the difficult decision to leave RippleMatch this year because I'm concerned about catastrophic risk from AI, and have been spending the last year thinking about ways to help. Given my background, I’ve been thinking a lot about for-profit ideas to help with alignment – many that can be VC-backed. Some of these ideas speak more directly to reducing catastrophic risk than others, but I think that all can put a founder in a strong position to help in the future.

Why I believe for-profit alignment orgs are valuable

I don’t think for-profit approaches are inherently better than building non-profits, pursuing government regulation, or other approaches, but I think that for-profit orgs can make a substantial impact while attracting a different pool of talent eager to work on the problem.

With VC dollars, a for-profit organization can potentially scale far more quickly than a non-profit. It could make a huge impact and not have its growth capped by donor generosity. As a result, there can be far more organizations working on safety in the ecosystem tapping into a different pool of resources. That said, any VC-backed company has a relatively low chance of success, so it’s a riskier approach.

Fundamentally, I believe that risk and compliance spend will grow extremely quickly over the coming decade, scaling with generative AI revenue. With comps in finance and cybersecurity, I’d guess that mid to high single digit percentages of overall AI spend will be on risk and compliance, which would suggest big businesses can be built here. Many startups tackling alignment will need to start by addressing short term safety concerns, but in doing so will position themselves to tackle long-term risks over time.

Onto the actual ideas!

Robustness approaches

Testing / benchmarking software

Test case management needs to look very different for LLMs compared to typical softw...