Bookmarks (705)

Published on June 1, 2025 2:29 PM GMTHi all — I’m helping organize a small economics unconference that I think will be exciting for rational-ish people. It's coming up soon (June 13th-16th), but we have a few more spots available. More details below!Second OrderDates: 13th to 16th June 2025Location: Abbey House, Audley End Estate, ~30min from Cambridge, UK✨ Apply here! Applications are due June 6th, EOD...

Published on May 31, 2025 10:10 PM GMTIn competitive systems, whether geopolitical, economic, technological, or memetic, a recurrent pattern emerges: actors willing or able to escalate tend to outperform those who restrain themselves. This article proposes a general principle to formalize that dynamic, examines its structural foundations, and discusses the fragility of mechanisms meant to suppress it.1. The Escalation Dominance Principle (EDP)I propose the...

Published on May 31, 2025 10:09 PM GMTEpistemic Status: Exploratory. I'm new to AI alignment research but have background in math and read psychotherapy texts extensively while spending two years as a ghost-writer. Seeking feedback to refine these connections.Tl;dr: I suggest therapeutic techniques from a variety of psychotherapeutic schools of thought can inspire new approaches to AI learning and alignment. I reinterpret three recent...

Published on June 1, 2025 11:48 AM GMTThis is a crosspost of a post from my blog, Metal Ivy. The original is here: An Opinionated Guide to Statistical Significance.I think for the general audience the value of this post is the first half, which tries to give a practical intuition for p-values.But for LessWrong the value is probably the mathematical second half, which breaks...

Published on June 1, 2025 8:18 AM GMTIn a previous article I detailed FSU Law professor Nadia Batenka's proposed "Inverted Sliding Scale Framework" approach to the question of legal personhood for digital intelligences. In this article, I am going to examine another paper approaching the issue which was first authored by Claudio Novelli, Giorgio Bongiovanni, and Giovanni Sartor. Since its original writing it has been...

Published on May 31, 2025 10:04 PM GMTI study Economics and Data Science at the University of Pennsylvania. I used o1-pro, o3, and Gemini Deep Research to expand on my ideas with examples, but have read and edited the paper to highlight my understanding improved on by AI. I. The AI Labor Debate: Beyond "Robots Taking Our Jobs"The Prevailing Narrative: Supply-Side AutomationThe discourse surrounding artificial...

Published on May 31, 2025 9:17 PM GMTAdapted from this twitter thread. See this as a quick take.Mitigation StrategiesHow to mitigate Scheming?Architectural choices: ex-ante mitigationControl systems: post-hoc containmentWhite box techniques: post-hoc detectionBlack box techniquesAvoiding sandbaggingWe can combine all of those mitigation via defense-in-depth system (like the Swiss Cheese model below)I think that applying all of those strategies should divide the risk by at least...

Published on May 31, 2025 6:20 PM GMTThere have recently been various proposals for mitigations to "the intelligence curse" or "gradual disempowerment"—concerns that most humans would end up disempowered (or even dying) because their labor is no longer valuable. I'm currently skeptical that the typically highlighted prioritization and interventions are best and I have some alternative proposals for relatively targeted/differential interventions which I think would be more...

Published on May 31, 2025 4:30 PM GMTThere’s a story — I'm at a conference and cannot access my library, but I believe it comes from Structural Anthropology by Claude Lévi-Strauss — about an anthropologist who took a member of an Amazonian tribe to New York.The man was overwhelmed by the city, but there were two things that particularly interested him.One was the bearded...

Published on May 31, 2025 4:18 PM GMTRational Animations takes a look at Tom Davidson's Takeoff Speeds model (https://takeoffspeeds.com). The model uses formulas from economics to answer two questions: how long do we have until AI automates 100% of human cognitive labor, and how fast will that transition happen? The primary scriptwriter was Allen Liu (the first author of this post), with feedback from...

Published on May 30, 2025 4:57 PM GMTThese are the most significant pieces of life advice/wisdom that have benefited me, that others often don't think about and I end up sharing very frequently. I wanted to share this here for a few reasons: First and most important, I think many people here will find this interesting and beneficial. I am very proud of this post...

Published on May 30, 2025 2:11 PM GMTStatus: musings. I wanted to write up a more fleshed-out and rigorous version of this, but realistically wasn't likely to every get around to it, so here's the half-baked version. Related posts: Firming Up Honesty Around Its Edge-Cases, Deep honesty What I mean by 'honesty' There are nuances to this, but I think a good summary is...

Published on May 30, 2025 1:46 PM GMTI envision a future more chaotic than portrayed in AI 2027. My scenario, the Rogue Replication Timeline (RRT), branches off mid-2026. If you haven’t already read the AI 2027 scenario, I recommend doing so before continuing.You can read the full version on my blog, but I have copied the summary and first half of the scenario below.This...

Published on May 30, 2025 10:50 AM GMTLetting kids be kids seems more and more important to me over time. Our safetyism and paranoia about children is catastrophic on way more levels than most people realize. I believe all these effects are very large: It raises the time, money and experiential costs of having children so much that many choose not to have children,...

Published on May 30, 2025 1:21 AM GMTThe Geometry of LLM Logits (an analytical outer bound) 1 Preliminaries Symbol Meaning d.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px...

Published on May 30, 2025 12:23 AM GMTFrom June 2-6, the Center for Applied Rationality will be running an experimental mini-workshop at Arbor Summer Camp. This is an in-person event held in Berkeley, CA at the Lighthaven campus and requires paid admission -- read on for more details.This workshop is "mini"This workshop will last for about three days (two half-days on the edges, two...

Published on May 29, 2025 10:02 PM GMTHello from the Center for Applied Rationality!Some of you may have attended our classic applied rationality workshops in the past; others of you may have wanted to attend a workshop but not yet had a chance to. It's been a while since we've last run public-facing workshops, but we wanted to say: We're not dead! (More on that...

Published on May 29, 2025 9:42 PM GMTIn previous posts in this sequence, I laid out a case for why most AI governance research is too academic and too abstract to have much influence over the future. Politics is noisy and contested, so we can’t expect that good AI governance ideas will spread on their own – we need a large team of people...

Published on May 29, 2025 6:55 PM GMTThis post benefitted greatly from comments, suggestions, and ongoing discussions with David Duvenaud, David Krueger, and Jan Kulveit. All errors are my own.A few months ago, I and my coauthors published Gradual Disempowerment (GD hereafter). It was mostly about how things might go wrong, but naturally a lot of the resulting interest has been about solutions. We have some...

Published on May 29, 2025 6:49 PM GMTEveryone around me has a notable lack of system prompt. And when they do have a system prompt, it’s either the eigenprompt or some half-assed 3-paragraph attempt at telling the AI to “include less bullshit”.I see no systematic attempts at making a good one anywhere.[1](For clarity, a system prompt is a bit of text—that's a subcategory of "preset"...

Published on May 28, 2025 6:30 PM GMTSince Claude 4 Opus things have been refreshingly quiet. Video break! The First Good AI Videos First up we have Prompt Theory, made with Veo 3, which I am considering the first legitimately good AI-generated video I’ve seen. It perfectly combining form and function. Makes you think. Here’s a variant, to up the stakes a bit, then...

Published on May 28, 2025 4:19 PM GMTIntroductionI have long been very interested in the limitations of LLMs because understanding them seems to be the most important step to getting timelines right. Right now there seems to be great uncertainty about timelines, with very short timelines becoming plausible, but also staying hotly contested. This led me to revisit LLM limitations and I think I noticed a...

Published on May 28, 2025 4:03 PM GMTOkay, I think it's time to stop polishing this playlist & post it. This is music about the moods of people I met at Manifest. I'm pretty delighted with it! Discuss

Published on May 28, 2025 4:00 PM GMTWelcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.In this edition: Google released a frontier video generation model at its annual developer conference; Anthropic’s Claude Opus 4 demonstrates the danger of relying on voluntary governance.Listen to the AI Safety Newsletter for free...

Published on May 28, 2025 12:57 PM GMTAn interim research reportSummaryWe introduce a novel methodology for quantitatively evaluating metacognitive abilities in LLMsWe present evidence that some frontier LLMs introduced since early 2024 - but not older or smaller ones - show some metacognitive abilitiesThe metacognitive abilities that current LLMs do show are relatively weak, and manifest in a context-dependent manner; the models often prefer...

Published on May 28, 2025 11:42 AM GMTDuring a visit to a Hong Kong children’s welfare home, I met a 12-year-old girl I'll call Kylie. She had suffered a severe illness that left her blind, deaf, non-verbal, and nearly immobile, yet no identified damage was done to her brain.The staff described her, without hesitation, as “always cheerful,” and indeed she smiled the entire time...

Published on May 28, 2025 10:50 AM GMTThe following analogy is intended to illustrate a novel proof-theoretic concept. It is metaphorical in nature and should not be interpreted literally.The Vanishing SkyConsider the following analogy:In a universe undergoing accelerating expansion, distant galaxies gradually slip beyond our horizon, until the point at which their light can no longer reach us. These galaxies do not disappear in...

Published on May 28, 2025 9:30 AM GMTContext: This is a linkpost for https://aisafety.info/questions/NM3I/6:-AI%E2%80%99s-goals-may-not-match-ours This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website. Making AI goals match our intentions is called the alignment problem.There’s some ambiguity in the term “alignment”. For example, when people talk about...

Published on May 28, 2025 9:30 AM GMTContext: This is a linkpost for https://aisafety.info/questions/NM3J/5:-AI-may-pursue-goals This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?One way to imagine these...

Published on May 28, 2025 6:21 AM GMTThis is a link-post for a new paper I read: Safety Pretraining: Toward the Next Generation of Safe AI by Pratyush Maini, Sachin Goyal, et al.For a couple of years I (and others) have been proposing an approach to alignment: what the authors of this recent paper name "safety pretraining". In a nutshell: that it's best to...

Published on May 26, 2025 6:29 PM GMTAs promised in the previous instalment on meter, let’s explore rhyming from a methodological perspective.The first difference between meter and rhyme lies in their opposite obviousness: the first one is subtle, requiring a learned and attuned ear; the second is so sonorous and clear that children hear it as self-evident.Take one of my favorite Robert Frost poems: ...

Published on May 26, 2025 6:26 PM GMTIn my experience, the most annoyingly unpleasant part of research[1] is reorganizing my notes during and (especially) after a productive research sprint. The "distillation" stage, in Neel Nanda's categorization. I end up with a large pile of variously important discoveries, promising threads, and connections, and the task is to then "refactor" that pile into something compact and well-organized,...

Published on May 26, 2025 6:20 PM GMTRishi Jha, Collin Zhang, Vitaly Shmatikov and John X. Morris published a new paper last week called Harnessing the Universal Geometry of Embeddings.Abstract of the paper (bold was added by me):We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised...

Published on May 26, 2025 5:38 PM GMTThe full post is long, but you can 80/20 the value with the 700 word summary! Over half the post is eight optional case studies. Thanks to Jemima Jones, Claude 4 Opus and Gemini 2.5 Pro for help copy-editing and draftingTL;DR: I recommend giving advice by asking questions to walk someone through key steps in my argument — often I’m...

Published on May 26, 2025 5:22 PM GMTAt NeurIPS 2024, Ilya Sutskever delivered a short keynote address in honor of his Seq2seq paper, published a decade earlier. It was his first—and so far only—public appearance to discuss his research since parting ways with OpenAI.The talk itself shed little light on his current work. Instead, he reaffirmed the prevailing view that the “age of pre-training”...

Published on May 26, 2025 5:29 PM GMTIn the last post I took the seemingly-naive stance that "pain is just information" and "can't actually be a problem", implying that painful situations can be dealt with without suffering by just looking through the pain towards the reality at which it points. I did give an example of how this allowed me to quickly resolve an...

Published on May 26, 2025 3:26 PM GMTHi! I'm a rising junior in undergrad, working on a cognitive science major with neuroscience and AI focuses, and I was hoping to get some advice/pointers on AI safety work. I'm interested in both the governance and technical sides, but my academic work slightly predisposes me to the latter. Any advice, help, ideas, links to other posts,...

Published on May 26, 2025 4:00 PM GMTI'm making a website on AI companies' model evals for dangerous capabilities: AI Safety Claims Analysis. This is approximately the only analysis of companies' model evals, as far as I know. This site is in beta; I expect to add lots more content and improve the design in June. I'll add content on evals, but I also tentatively...

Published on May 26, 2025 4:00 PM GMTThe new scorecard is on my website, AI Lab Watch. This replaces my old scorecard. I redid the content from scratch; it's now up-to-date and higher-quality. I'm also happy with the scorecard's structure: you can click on rows, columns, and cells and zoom in to various things. Check it out! Thanks to Lightcone for designing the site.While it is...

Published on May 26, 2025 3:07 PM GMTThe chickenpox virus waited for decades, attacking the moment my immune system wobbled.[1] It advanced down my nerves, spreading blisters along its path. Known as shingles, this kind of viral attack is generally considered a very bad time.The blisters make nerves go haywire. They start sending chaotic signals back to the brain, jagged and dissonant.[2] The brain struggles to...

Published on May 25, 2025 6:01 PM GMTOne of the best and easiest ways to influence a corporation is to own it. Google offers both $GOOG, class C non-voting, and $GOOGL, class A voting. If you especially don’t care about other issues, a small portion of the voting owner base caring about advanced AI issues can strongly affect how GDM operates.Discuss

Published on May 25, 2025 5:54 PM GMTI posted a quick take that advocacy may be more effective than direct donation to alignment research.  I am not an AI researcher and I'm not an influencer, so I'm not well positioned to do either.  I see on the "How can I help" FAQ that there are options to donate, but they look like donating to...

Published on May 25, 2025 5:04 PM GMT09:46: Everyone wants to be close to everyone else to do good business effectively. Like Silicon Valley for example, smart people want to go there and be close to other smart people in order to start a successful business.09:47: But as soon as they get close to each other, imagine that they all have to pay this...

Published on May 25, 2025 2:00 PM GMTUnlike everyone else, Anthropic actually Does (Some of) the Research. That means they report all the insane behaviors you can potentially get their models to do, what causes those behaviors, how they addressed this and what we can learn. It is a treasure trove. And then they react reasonably, in this case imposing their ASL-3 safeguards on...

Published on May 25, 2025 12:58 PM GMTEpistemic Status: Over years of reading alignment plans and studying agent foundations, this is my first serious attempt to formulate an alignment research program that I (Cole Wyeth) have not been able to find any critical flaws in. It is far from a complete solution, but I think it is a meaningful decomposition of the problem into...

Published on May 25, 2025 12:00 PM GMTLessons from shutting down institutions in Eastern Europe.This is a cross post from: https://250bpm.substack.com/p/meditations-on-doge Imagine living in the former Soviet republic of Georgia in early 2000’s:All marshrutka [mini taxi bus] drivers had to have a medical exam every day to make sure they were not drunk and did not have high blood pressure. If a driver did not...

Published on May 24, 2025 8:05 PM GMTThe purpose of this post is to argue for prioritizing the 'democratizing'  of AGI, even before we've figured out alignment.It's also an argument against favoring historically useful economic and social policies in the runup to a post AGI world.The intended effect is to encourage people to start seriously discussing and organizing ways to increase the size of...

Published on May 25, 2025 5:40 AM GMTSimulators was posted two and a half years ago and quite a bit has happened since then.  Ideas that were once thought experiments now exist as observable data points.  This post will sample a few significant findings to see how the simulator and agent frames hold up.  The selection criteria for papers to review was arbitrary, basically:...

Published on May 24, 2025 9:24 PM GMTI have noticed that there are talks around about moral ASI[1]. And I think that to use the word "moral" in relation to Artificial Intelligence, we must be absolutely confident in our knowledge of how morality works for human beings. Otherwise, we must avoid using such combinations of words to avoid anthropomorphic bias.The question of morality in...

Published on May 24, 2025 8:52 PM GMTThere has recently been a back-and-forth over Claude 4 Opus:Anthropic: Opus can help people make chemical weapons!Also Anthropic: Don't worry, it's mitigated!Day 1 (2?) Jailbreakers: Lol, lmaoWhere the Jailbreakers think the info they got out of the model would make it materially easier to produce sarin gas. Although I do note that not one but two synthesis...