trentmkelly/LessWrong-43k huggingface dataset:
Consciousness is irrelevant - instead solve alignment by asking this question
by Oliver Siegel — 2023-03-04T22:06:46.424Z

I've noticed that there are 2 flavors of the alignment problem. One is about the technical how to and practical engineering, the other is about humanities, social sciences, human behavior and psychology.

What they both have in common is doomsday scenarios about paperclip maximizers.

I don't know about you, but personally I don't care if an unaligned AI is aware of its experience while turning the universe into a paperclip, or if the whole thing is happening entirely without consciousness. What i care about is: How can we make AI smart enough to prevent that from happening?

That said, I find especially in amateur circles that CONSCIENCE is not just omitted from the discussion, it's actually poorly understood.

Arguably, if we find an algorithm for empathy, we can solve the alignment problem rather easily. Humans seem to be capable of monitoring their actions for morality and preventing harm to others, how hard can it be to build an artificial brain that can do the same?

How would you explain to someone what the difference is between consciousness and conscience?
More money with less risk: sell services instead of model access
by lemonhope — 2023-03-04T20:51:36.480Z

OpenAI is currently charging 100,000 times less per line of code than professional US devs.[1]

An LLM's code output is of course less reliable than a professional's. And it is hard to use a text-completion API effectively in large projects.

What should you do if you've got a model on your hands that solves those problems?

You could operate as a software development company. They tend[2] to charge $100-200k for simple mobile apps and there's basically no ceiling on the cost for complex apps over their lifetime. Devs make up the majority of a normal firm's personnel and costs; coding takes most of the app development time; bugs in code are one of the primary sources of project extension and failures. By using your model you can make better software, complete it faster, succeed more often, charge a lower price, and make a higher profit.

Going further, if you've really got a good model, then you can do very well by building competitors to adobe products, salesforce products, SAP products, google search, mongodb, etc.

Someone who has a build-anything machine would be a fool to sell a cheap build-anything service instead of using it themselves and selling the result. Particularly because selling the general service directly is likely to encourage and inspire copycats, including open-source ones who will delete your market. If it really builds the entire thing then you'll probably also be liable for negative consequences, which again have no ceiling.

Fewer risks, big and small

Some common misuse risks you can avoid/reduce (and eliminate associated liability):

Someone tricks your API into doing something awful and pastes it into a tweet

Spam generation for political campaigns, cryptocurrencies, etc

Common hacking ("write a test to see if my server has a log4j vulnerability")

Targeted manipulation and spearphishing

Larger risks you can avoid/reduce:

Your incredible model motivates countless AI researchers. People reverse-engineer some of the architecture in online discussions. The state of the art is quickly advanced. We have less time to prepare for strong general AI.

Hackers steal your model weights (if you don't advertise your model then you'll attract less attention from hackers)

People try to get your model to act like an agent and copy itself around. They succeed. You have no way of shutting it down or monitoring what it is doing.

Someone tries to get your mod

...
Contra "Strong Coherence"
by DragonGod — 2023-03-04T20:05:28.346Z

Polished from my shortform

See also: Is "Strong Coherence" Anti-Natural?

Introduction

Many AI risk failure modes imagine strong coherence/goal directedness[1] (e.g. [expected] utility maximisers).
Such strong coherence is not represented in humans (or any other animal), seems unlikely to emerge from deep learning and may be "anti-natural" to general intelligence in our universe[2][3].

I suspect the focus on strongly coherent systems was a mistake that set the field back a bit, and it's not yet fully recovered from that error[4].
I think most of the AI safety work for strongly coherent agents (e.g. decision theory) will end up inapplicable/useless for aligning powerful systems, because powerful systems in the real world are "of an importantly different type".

Ontological Error?

I don't think it nails everything, but on a purely ontological level, @Quintin Pope and @TurnTrout's shard theory feels a lot more right to me than e.g. HRAD. HRAD is based on an ontology that seems to me to be mistaken/flawed in important respects.

The shard theory account of value formation (while lacking) seems much more plausible as an account of how intelligent systems develop values (where values are "contextual influences on decision making") than the immutable terminal goals in strong coherence ontologies. I currently believe that (immutable) terminal goals is just a wrong frame for reasoning about generally intelligent systems in our world (e.g. humans, animals and future powerful AI systems)[2].

Theoretical Justification and Empirical Investigation Needed

I'd be interested in more investigation into what environments/objective functions select for coherence and to what degree said selection occurs.
And empirical demonstrations of systems that actually become more coherent as they are trained for longer/"scaled up" or otherwise amplified.

I want advocates of strong coherence to explain why agents operating in rich environments (e.g. animals, humans) or sophisticated ML systems (e.g. foundation models[5]) aren't strongly coherent.
And mechanistic interpretability analysis of sophisticated RL agents (e.g. AlphaStar, OpenAI Five [or replications thereof]) to investigate their degree of coherence.

Conclusions

Currently, I think strong coherence is unlikely (plausibly "anti-natural"[3][2]) and am unenthusiastic about research agendas and threat models predicated on strong coherenc...
The Practitioner's Path 2.0: A new framework for structured self-improvement
by  — 2023-03-04T19:19:04.153Z
Link: https://guildoftherose.org/articles/a-new-dawn-for-structured-self-improvement

The Guild of the ROSE is proud to announce the full launch of the Practitioner's Path 2.0, a new framework for structured self-improvement.

Two years ago, we founded the Guild to provide structure and community for people interested in self-improvement. Since then, we've realized that our old strategy of weekly workshops and community groups needed a revision. People liked our workshops, but more often than not, they struggled to translate abstract ideas into tangible life improvements.

One idea that had floated around the Guild since its creation was the dream of a Skilltree for life — what if self-improvement was as easy as leveling up a new videogame character?

The Practitioner's Path 2.0 is our attempt to make that dream a reality. Self improvement is ultimately rewarding, but that doesn't mean it's easy. Our goal is to design systems that will remove the barriers to self improvement, whether by reducing the cognitive load of wondering what to work on, providing extrinsic social motivation, or playing into a completionist challenge-and-reward mindset.

This article introduces the Practitioner's Path 2.0, starting with the three aspects of the Path:

Skills: discrete blocks of achievement
Attributes: long-term, concrete metrics
Tasks: specific goals you want to accomplish

Afterwards, we'll briefly discuss the future role of workshops in the Guild and wrap up with some changes to our pricing model.

Part 1: the Skilltree

The heart of the Path 2.0 is the Skilltree. The Skilltree is one part map, one part achievement log, and one part reputation system.

A Map for Self-Improvement

Each Skill in the Path has a certain number of prerequisites. For example, Basic Life Enhancement I (fixing problems in your life) requires you to acquire Unseen Chances I (identify problems in your life) first:

Lines and arrows indicate prerequisites.

The example above is simple, but it illustrates the power of having your self-improvement path laid out before you. Rather than trying to figure out an optimal path (and potentially choice-paralyzing yourself into inaction), you can simply look at the Skilltree and work on the next Skill available.

The three states of a Skill.

The style of a Skill indicates whether:

Unlit: you're missing prerequisites (unlit)
Dim: you can work on it next (dim)
Bright + double border: you've acquired it

An Achievement Log

Every time you acquire a Skill, it's permanently assoc...
The Benefits of Distillation in Research
by Jonas Hallgren — 2023-03-04T17:45:22.547Z

Epistemic status: I wrote this six months ago, but I never felt satisfied with it, so I kind of forgot about it. At this point, I'll just release it. It is similar to Chris Olah's article on research debt, but it has a different perspective on presenting the mechanisms.

(The examples in this post are not made to be serious proposals)

This post is meant to bring forward the causal mechanisms of clearer communication and the potential of “Distillations”. Distillation here will be defined as the process of re-organising existing information and ideas into a new perspective. The post is partly meant to increase the clarity of what a distillation might be, and partly meant to give some more insight into how it may speed up research.

A model of analogous idea generation in AI safety

This is a simple model of the forefront of research in the AI safety field:

The frontier gets forward by different agendas making progress in each of their own domains. However, just like in battle frontiers, communication is very important as if we don’t use the progress other domains make, we might waste a lot of time and do a lot of double work.

For example, if we're working on infrabayesianism and trying to solve a complicated problem regarding ontology verification (not sure if this is an actual thing being worked on in infrabayesianism). We then try to take inspiration from ELK, as it has been worked on a lot in the reference class of ontologies. This might look something like this:

Figure 2: Progress in alignment

ELK has a research stack of 7 different perspectives in our fictional example. The red blocks are perspectives that don’t yield any light on the problem, whilst the blue lens helps with progress on the problem. A visualization of this is the following:

Figure 3: Very simplified perspectives on a problem

Here we can see the old perspective in orange, the old framing of the problem. In our fictional example, the current orange infrabayesian frame might sound something like “This is a one-to-many problem where we have to apply causal inference to back-chain, to formalise this we use causal scrubbing with Knightian uncertainty”

Whilst the new blue approach from the ELK direction might sound something like “If we have a neural network that could spot whenever a matrix describing the original NN internals is changed too much, then we could theoretically spot distribution shifts”

Th...
Optimal Music Choice
by mbazzani — 2023-03-04T17:26:57.915Z

I've been thinking whether it would be possible to optimize music choice, and it seems to me there are a few relevant questions. I'd be happy to hear some thoughts or related research on this.

Is there a way to do randomized control trials of music? It seems like selection effects are inevitable, especially since individual engagement/attitude may be important for musical engagement. The only way I can think of circumventing this is something like lobby music, so that people are neither in charge of their music choices, but also don't feel forced into the choice.
How do effects of music change for different combinations of listener moods and music styles? For example, listening to sad music might not make you sadder if you listen to it in a good mood, and it might give life a greater sense of meaning. Personally, I've noticed that listening to music that matches my mood frequently creates a vicious cycle.
Optimal spacing/variety? I've personally noticed that I get very giddy when I re-discover a song I know well but haven't listened to in a long time. Perhaps active song choice/browsing is slightly better for enjoyment than playlists?

Also, this is my first post, so I'd really appreciate any tips on style, content, or posting norms that I can improve on. Thanks!
Why don't more people talk about ecological psychology?
by Ppau — 2023-03-04T17:03:17.588Z

An intermediate step in the abstraction staircase

I can't claim to have deep knowledge or understanding about the topic, but ecological dynamics seems like a really interesting and underrated approach, and nobody else has done it, so I'll do my best.

If you've never heard of ecological psychology before, it is mostly interested in coaching, sports science, driving, what one might call more "system 1" stuff. I personally discovered it through Rob Gray's excellent Perception Action podcast and website, which are focused on these topics.

The term "ecological" comes from the idea that the behavior of an agent can be explained entirely through their environment (broadly speaking). More explicitly, ecological psychologists do their absolute best to avoid using concepts such as beliefs, mental models of the world, computation, memories and so on. This may seem completely incompatible with the Bayesian mindset, with its credence-sprinkled map of the territory, and indeed, partisans of the ecological approach often get involved in heated debates against predictive processing or similar approaches. They also use a lot of special terms and definitions ("field of affordances" "action manifolds" "prospection not prediction") that may seem pedantic and willfully obscure, and it took me a while to get the way they think. But in my view, their ideas are more compatible with Bayesianism that they might seem, and they mesh really well with lesswrongian rationalism.

For instance, I think the argument against using concepts about about internal states such as credences or memories can be seen as a game of rationalist taboo, meant to avoid explaining things away without understanding them. Let's say we want to know how an athlete can move in the right way to catch a ball. A common lay answer might be "well, the athlete elaborates a model of where the ball will be in the future, and then they run and move their hand to the right spot according to that model". This may seem both satisfactory and obvious, but:

If you think about it, what question about behavior *can't* you answer by “the agent did this action because they modeled the world and according to their models this action was the best"? If you can explain everything...
If you analyze behavior in that way, you're going to miss some very nice insights. In the example of running to catch a ball (the "outfielder problem"

...
Switching to Electric Mandolin
by jefftk — 2023-03-04T15:40:05.218Z

A little over a year ago I bought a

solid-body
electric mandolin. At the time I was thinking I would still
mostly play my

acoustic or bring both to
gigs, but instead I've ended up switching entirely. I

compared recordings and

amplification

options, and it turns
out that the cleaner sound works much better in the chaos of a dance
hall. I also really like that it works

with a talk box and that I don't
need to worry about feedback on stage anymore.

I still play the acoustic when I'm jamming with friends or family, but
it now spends most months without getting out of its case. Which does
make me sad; it's a good mandolin and I enjoy playing it.

Comment via: facebook
Predictive Performance on Metaculus vs. Manifold Markets
by nikos — 2023-03-04T08:10:40.941Z

(crossposted from the EA Forum)

TLDR

I analysed a set of 64 (non-randomly selected) binary forecasting questions that exist both on Metaculus and on Manifold Markets.
The mean Brier score was 0.084 for Metaculus and 0.107 for Manifold. This difference was significant using a paired test. Metaculus was ahead of Manifold on 75% of the questions (48 out of 64).
Metaculus, on average had a much higher number of forecasters
All code used for this analysis can be found here.

Conflict of interest note
I am an employee of Metaculus. I think this didn't influence my analysis, but then of course I'd think that and there may be things I haven't thought about.

Introduction

Everyone likes forecasts, especially if they are accurate (well, there may be some exceptions). As a forecast consumer the central question is: where should you go to get your best forecasts? If there are two competing forecasts that slightly disagree, which one should you trust most?

There are a multitude of websites that collect predictions from users and provide aggregate forecasts to the public. Unfortunately, comparing different platforms is difficult. Usually, questions are not completely identical across sites which makes it difficult and cumbersome to compare them fairly. Luckily, we have at least some data to compare two platforms, Metaculus and Manifold Markets. Some time ago, David Glidden created a bot on Manifold Markets, the MetaculusBot, which copied some of the questions on the prediction platform Metaculus to Manifold Markets.

Methods

Manifold has a few markets that were copied from Metaculus through MetaculusBot. I downloaded these using the Manifold API and filtered for resolved binary questions. There are likely more corresponding questions/markets, but I've skipped these as I didn't find an easy way to match corresponding markets/questions automatically.
I merged the Manifold markets with forecasts on corresponding Metaculus questions. I restricted the analysis to the same time frame to avoid issues caused by a question opening earlier or remaining open longer on one of the two platforms.
I compared the Manifold forecasts with the community prediction on Metaculus and calculated a time-averaged Brier Score to score forecasts over time. That means, forecasts were evaluated using the following score: S(p,t,y)=∫Tt0(pt−y)2dt, with resolution

...
Contra Hanson on AI Risk
by Liron — 2023-03-04T08:02:02.375Z

Robin Hanson wrote a new post recapping his position on AI risk (LW discussion). I've been in the Eliezer AI-risk camp for a while, and while I have huge respect for Robin’s rationality and analytical prowess, the arguments in his latest post seem ineffective at drawing me away from the high-doom-worry position.

Robin begins (emphasis mine):

First, if past trends continue, then sometime in the next few centuries the world economy is likely to enter a transition that lasts roughly a decade, after which it may double every few months or faster, in contrast to our current fifteen year doubling time. (Doubling times have been relatively steady as innovations are typically tiny compared to the world economy.) The most likely cause for such a transition seems to be a transition to an economy dominated by artificial intelligence (AI). Perhaps in the form of brain emulations, but perhaps also in more alien forms. And within a year or two from then, another such transition to an even faster growth mode might plausibly happen.

And adds later in the post:

The roughly decade duration predicted from prior trends for the length of the next transition period seems plenty of time for today’s standard big computer system testing practices to notice alignment issues.

Robin is extrapolating from his table in Long-Term Growth As A Sequence of Exponential Modes:

I get that there’s a trend here. But I don’t get what inference rule Robin's trend-continuation argument rests on.

Let’s say you have to predict whether dropping a single 100-megaton nuclear bomb on New York City is likely to cause complete human extinction. (For simplicity, assume it was just accidentally dropped by the US on home soil, not a war.)

As far as I know, the most reliably reality-binding kind of reasoning is mechanistic: Our predictions about what things are going to do rest on deduction from known rules and properties of causal models of those things.

We should obviously consider the causal implications of releasing 100 megatons worth of energy, and the economics of having a 300-mile-wide region wiped out.

Should we also consider that a nuclear explosion that decimates the world economy would proceed in minutes instead of years, thereby transitioning our current economic regime much faster than a decade, thus violating historical trends? I dunno, this trend-breaking seems totally irrelevant to the question of w...
Bite Sized Tasks
by Johannes C. Mayer — 2023-03-04T03:31:30.404Z

Have tiny projects that are completable in one day. Instead of writing one long article about something, write a short blog post that only talks about some part of the topic, such that you can write and post it in one day. I hypothesize that this way, it is easier to get a reward signal because you get the "I am done" thing.

Very often I write up a blog post for hours, but then stop because it is too late. I almost never come back to these posts. Then it feels like I did not accomplish anything (in fact I have not, in terms of having new posts up). Almost all the posts that I have on LessWrong are actually done in just one day.

The general strategy of breaking down your task into small subtasks, that can be completed in less than 1 hour, is a general strategy that I have heard come up many times. E.g. in game dev. Decomposing a project into tasks that can be done in a short amount of time is probably important to get a reward signal. My guess is that this also helps with the planning fallacy. Decomposing projects into subprojects, each composed of smaller tasks, where each subproject can be done in just one day, might be a good extension of this strategy.

I guess the problem is that when I am writing something, it is a lot harder to decompose my writing into tasks that can be completed in a short period of time. Normally I have as the task "write and post this", which I then never do if it takes more than one day. It is just kind of hard to pick up where I left off. This is amplified by the fact that I often don't know what I am gonna get out of writing something. Often I write to understand something better myself.

So I want to try to write my blog posts such that I can finish them in one day. That means that I should be stricter in where I set borders for a post, and possibly even stop early, and leave the rest of the topic to a future blog post.

I now realize that there are probably many other ways to tackle this problem. Basically, I failed at holding off on proposing solutions here, and just came up with one. But looking at the problem, I realize that the underlying problem is that I do not come back to working on posts at later points in time. Doing a post in one day might be a good solution, but there are many other things that might work that I did not think about, because I was anchored to this solution.

For example, setting up some routine, where I ha...