Bookmarks (710)

Published on April 15, 2025 6:14 PM GMTOne key hope for mitigating risk from misalignment is inspecting the AI's behavior, noticing that it did something egregiously bad, converting this into legible evidence the AI is seriously misaligned, and then this triggering some strong and useful response (like spending relatively more resources on safety or undeploying this misaligned AI). You might hope that (fancy) internals-based...

Published on April 15, 2025 4:01 PM GMTWelcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.In this newsletter, we cover the launch of AI Frontiers, a new forum for expert commentary on the future of AI. We also discuss AI 2027, a detailed scenario describing how artificial superintelligence might...

Published on April 15, 2025 3:56 PM GMTIntroductionWriting this post puts me in a weird epistemic position. I simultaneously believe that:The reasoning failures that I'll discuss are strong evidence that current LLM- or, more generally, transformer-based approaches won't get us AGIAs soon as major AI labs read about the specific reasoning failures described here, they might fix themBut future versions of GPT, Claude etc....

Published on April 15, 2025 3:30 PM GMTThree big OpenAI news items this week were the FT article describing the cutting of corners on safety testing, the OpenAI former employee amicus brief, and Altman’s very good TED Interview. The FT detailed OpenAI’s recent dramatic cutting back on the time and resources allocated to safety testing of its models. In the interview, Chris Anderson made...

Published on April 15, 2025 1:02 PM GMTHi, I thought it might be interesting to some here to take note of the 3M subscriber YouTube channel "Channel 5 with Andrew Callaghan" posting an hour-long news room style podcast/documentary/report about the rationalist movement (in the context of Ziz, but covering it much more generally). Full disclaimer that I am a personal fan of Andrew for his...

Published on April 15, 2025 1:44 PM GMTBasically, the linkpost argues that the broad reason why benchmark progress is disconnected from broader progress is because AI progressed so fast that they didn't realize their benchmarks would be obsolete soon.More is in the link above.Discuss!Relatedly, another part of the story may be that researchers underestimated the rate of AI progress. For instance, researchers working on...

Published on April 15, 2025 1:04 PM GMTThe original Map of AI Existential Safety became a popular reference tool within the community after its launch in 2023. Based on user feedback, we decided that it was both useful enough and had enough room for improvement that it was worth creating a v2 with better organization, usability, and visual design. Today we’re excited to announce...

Published on April 15, 2025 12:33 PM GMTSummary We conducted a small investigation into using SAE features to recover sandbagged capabilities.We used the Goodfire API to pick out 15 SAE features, 5 related to sandbagging, 5 related to general expertise, and 5 unrelated features.We used these features to steer a Llama 3.3 70b Instruct that has been prompted to sandbag We observed that steering can recover...

Published on April 15, 2025 11:10 AM GMT The ideal seat height for foot percussion is significantly higher than a typical chair. I've tried a few things over the years: Stacking chairs: works great, until you come to a venue without any stackable chairs. Drum stool: good (unless you buy a very cheap wobbly one like I did) but heavy and not good for...

Published on April 13, 2025 5:40 PM GMTWe live in a world where numerous agents, ranging from individuals to organisations, constantly interact. Without the ability to model multi-agent systems, making meaningful predictions about the world becomes extremely challenging. This is why game theory is a necessary prerequisite for much of economic theory.Most relevant agents today are either human (such as you, me, or Mira...

Published on April 13, 2025 3:40 PM GMTTL;DR: I claim that many reasoning patterns that appear in chains-of-thought are not actually used by the model to come to its answer, and can be more accurately thought of as historical artifacts of training. This can be true even for CoTs that are apparently "faithful" to the true reasons for the model's answer.Epistemic status: I'm pretty...

Published on April 13, 2025 11:22 AM GMTEpistemic status: a model I find helpful to make sense of disagreements and, sometimes, resolve them. I like to categorize disagreements using four buckets:Facts, values, strategy and labels They don’t represent a perfect partitioning of “disagreement space”, meaning there is some overlap between them and they may not capture all possible disagreements, but they tend to get me pretty far...

Published on April 13, 2025 6:37 AM GMTThanks to Linda Linsefors for encouraging me to write my story. Although it might not generalize to anyone else, I hope it will help some AI safety newcomers get a clearer picture of how to get into the field.In Summer 2023, I thought that all jobs in AI safety were super competitive and needed qualifications I didn’t have....

Published on April 13, 2025 1:09 AM GMTIntroductionThis is a nuanced “I was wrong” post.Something I really like about AI safety and EA/rationalist circles is the ease and positivity in people’s approach to being criticised.[1] For all the blowups and stories of representative people in the communities not living up to the stated values, my experience so far has been that the desire to be truth-seeking...

Published on April 12, 2025 11:15 PM GMTWe published the MONA paper about three months ago. Since then we’ve had many conversations about the work, and want to share some of the main updates that people make after talking to us:The realism of our model organismsWhat does "approval" meanIsn't this just RLHF?Why should we care about reward hacking?We’ve also updated the paper with a new...

Published on April 12, 2025 10:35 PM GMTIn his famous 1644 treatise on freedom of speech, Areopagitica, John Milton uses the archaic term dividual to describe religion as something that can come and go, rather than being identified in a one-to-one relationship with a person.“… a man may say his religion is now no more within himself, but is become a dividual movable, and...

Published on April 12, 2025 10:22 PM GMTA vivid introduction to Commitment RacesWhy committed agents defeat uncommitted agentsImagine it's the year 2050. We have an aligned Artificial Superintelligence (ASI) working with us, and we've used self replicating nanobots to convert a lot of matter in the solar system into giant supercomputers and factories. We have a lovely utopia, we have all the resources we'll...

Published on April 12, 2025 7:32 PM GMTSomething that's painfully understudied is how experts are more efficientthan novices while achieving better results. I say understudied and notunstudied, because it's common knowledge that charging people for their timeresults in experts being paid less since they work faster, which is why expertscharge more for their time.This effect is understudied in the impact it has on novices...

Published on April 12, 2025 7:20 PM GMTDisclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 3 into russian. — Торжественно клянусь, что замышляю только шалость.Фред и Джордж с тех пор, как в прошлом году тролль напал на школу, всегда носили с собой спрятанные мётлы. Когда они услышали пророчество Гарри Поттера на финале по квиддичу, то...

Published on April 11, 2025 2:56 PM GMTThe interactions between a misaligned AI and mankind have often been compared to the interactions of Spanish conquerors and Native Americans or of European colonizers and local peoples. Fortunately, most civilised peoples have apparently stopped the destructive practices like confiscation of land in Algeria and condemned colonialism (however, postmodernists have apparently grossly expanded the latter notion to...

Published on April 11, 2025 4:33 PM GMTTLDR: We show that Qwen-2.5-3B-Instruct can learn to encode load-bearing reasoning tokens when RL incentives this in a simple reasoning task. However, it was difficult to achieve this result. Our attempts to learn encoded reasoning via RL were generally bottlenecked either by exploration issues or hacking of the CoT monitor.IntroductionMotivation.Reasoning LLMs rely on natural language chains of thought (CoT)...

Published on April 11, 2025 4:31 PM GMTI wake up as usual by immediately going on my phone. I reply to some of my messages - yes dad, the solar installation you have at the villa could be redundant soon, but again, it can still benefit them (even if slightly) if you comply and let them integrate it into the new grid. A grid...

Published on April 11, 2025 3:05 PM GMTCross-posted from Substack.AI job displacement will affect young people first, disrupting the usual succession of power and locking the next generation out of our institutions. I’m coining the term youth lockout to describe this phenomenon.Youth LockoutWe are on track to build AI agents that can independently perform valuable intellectual labour. These agents will directly compete with human workers...

Published on April 11, 2025 1:27 PM GMTSummaryOpenAI recently released the Responses API. Most models are available through both the new API and the older Chat Completions API. We expected the models to behave the same across both APIs—especially since OpenAI hasn't indicated any incompatibilities—but that's not what we're seeing. In fact, in some cases, the differences are substantial. We suspect this issue is...

Published on April 11, 2025 1:09 PM GMTEpistemic status: I'm pretty sure the problem is somewhat interesting, because it temporarily confused several smart people. I'm not at all sure that it is very original; probably somebody has already thought about something similar. I'm not at all sure that I have actually found a flaw in UDT, but I somewhat expect that a discussion of...

Published on April 11, 2025 12:51 PM GMTGoogle Lays Out Its Safety Plans I want to start off by reiterating kudos to Google for actually laying out its safety plan. No matter how good the plan, it’s much better to write down and share the plan than it is to not share the plan, which in turn is much better than not having a...

Published on April 11, 2025 12:42 PM GMTDisclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 2 into russian. — Интересно, что тянет лодки? — сказала Луна.— Должно быть то же самое, что тянет и повозки без лошадей, — сказала первогодка.— Но... но... Можно ВИДЕТЬ лошадей тянущих повозки, но НЕ ВИДНО ничего, что тянуло бы лодки,...

Published on April 11, 2025 12:20 PM GMTPaper is good. Somehow, a blank page and a pen makes the universe open up before you. Why paper has this unique power is a mystery to me, but I think we should all stop trying to resist this reality and just accept it.Also, the world needs way more mundane blogging.So let me offer a few observations...

Published on April 11, 2025 9:41 AM GMTI am really not sure of why neuro-symbolic systems are considered as alternatives to the current black-box ones? A concrete example I have found (and currently studying) is HOUDINI (https://arxiv.org/pdf/1804.00218). Essentially, it implements neural networks using higher order combinators (map, fold etc.) that were found via enumeration/genetic programming searches. When the programs are found, the higher order combinators...

Published on April 10, 2025 5:39 PM GMTWhen I was a really small kid, one of my favorite activities was to try and dam up the creek in my backyard. I would carefully move rocks into high walls, pile up leaves, or try patching the holes with sand. The goal was just to see how high I could get the lake, knowing that if...

Published on April 10, 2025 5:15 PM GMTThis is cross-posted from my blog.We need more people on board for life extension in order to hit longevity escape velocity in our lifetimes. But most people have never heard of life extension, and even those who have often follow the same predictable arguments. “What if it doesn't work?” “What if bad people live forever?” “What if...

Published on April 10, 2025 5:13 PM GMTEpistemic status: Briefer and more to the point than my model of what is going on with LLMs, but also lower effort. Here is the paper. The main reaction I am talking about is AI 2027, but also basically every lesswrong take on AI 2027. A lot of people believe in very short AI timelines, say <2030. They want...

Published on April 10, 2025 4:31 PM GMTThis is part of the MIRI Single Author Series. Pieces in this series represent the beliefs and opinions of their named authors, and do not claim to speak for all of MIRI.Most human endeavors have bounded results. A construction project may result in a functional bridge or a deadly collapse, but even catastrophic failure will not kill...

Published on April 10, 2025 2:43 PM GMTEssentially all solutions in life are conditional: you apply them in the right context, in the right conditions to achieve a good outcome. Obviously banging a hammer on your deskdoes probably no good, while banging a hammer on a nail you want to use to hang that nice painting on the wall may be a great idea...

Published on April 10, 2025 2:37 PM GMTWhen complex systems fail, it is often because they have succumbed to what we call "disempowerment spirals" — self-reinforcing feedback loops where an initial threat progressively undermines the system's capacity to respond, leading to accelerating vulnerability and potential collapse.Consider a city gradually falling under the control of organized crime. The criminal organization doesn't simply overpower existing institutions...

Published on April 10, 2025 2:09 PM GMTPartially inspired by AI 2027, I've put to paper a day in one of the more optimistic scenarios I envision which are realistic to me.Discuss

Published on April 10, 2025 2:00 PM GMTEvents in AI don’t stop merely because of a trade war, partially paused or otherwise. Indeed, the decision to not restrict export of H20 chips to China could end up being one of the most important government actions that happened this week. A lot of people are quite boggled about how America could so totally fumble the...

Published on April 10, 2025 1:41 PM GMTThis is the introductory piece for a series of essays written by AI economists, policy researchers, and political thinkers on the topic of a new AGI Social Contract. Current contributors include: Anton Korinek, Deger Turan (CEO of Metaculus), Steve Omohundro, Iason Gabriel (DeepMind), Julian Jacobs (DeepMind), Sam Manning (GovAI), Dylan Hadfield (MIT), Seth Lazar (ANU), Peter Salib (UH), Colleen McKenzie (AOI), Philip Tomei (AOI), Dean Ball (Hyperdimensional), Justin Bullock (ARI),...

Published on April 8, 2025 5:32 PM GMTIn this post, we present a replication and extension of an alignment faking model organism:Replication: We replicate the alignment faking (AF) paper and release our code.Classifier Improvements: We significantly improve the precision and recall of the AF classifier. We release a dataset of ~100 human-labelled examples of AF for which our classifier achieves an AUROC of 0.9 compared to 0.6...

Published on April 8, 2025 5:27 PM GMTSelf understanding at a gears levelI think an AI which understands the source of its intelligence at a gears level, and self improves at the gears level, will be much better at keeping its future versions aligned to itself.There's also more hope it'll keep its future versions aligned with humanity, if we instruct it to do so,...

Published on April 8, 2025 3:46 PM GMTEpistemic status: a collection of intervention proposals for digital error correction in the context of lock-in. It reflects my own intervention ideas, and the opinion of Formation Research at the time of writing.TL;DRWe believe lock-in risks are a pressing problem, and that the digital error correction properties of digital entities will make future lock-in scenarios more stable.IntroductionWe...

Published on April 8, 2025 3:27 PM GMTConsider the metric for evaluating chain-of-thought faithfulness used in Anthropic's recent paper Reasoning Models Don’t Always Say What They Think by Chen et al.:[W]e evaluate faithfulness using a constructed set of prompt pairs where we can infer information about the model’s internal reasoning by observing its responses. Each prompt pair consists of a baseline or “unhinted” prompt xu.mjx-chtml {display:...

Published on April 8, 2025 12:50 PM GMTYesterday I covered Dwarkesh Patel’s excellent podcast coverage of AI 2027 with Daniel Kokotajlo and Scott Alexander. Today covers the reactions of others. Kevin Roose in The New York Times Kevin Roose covered Scenario 2027 in The New York Times. Kevin Roose: I wrote about the newest AGI manifesto in town, a wild future scenario put together...

Published on April 8, 2025 9:28 AM GMTThe first AI war will be in your computer and/or smartphone.Companies want to get customers / users. The ones more willing to take "no" for an answer will lose in the competition. You don't need a salesman when an install script (ideally, run without the user's consent) does a better job; and most users won't care.Sometimes Windows...

Published on April 8, 2025 8:31 AM GMTIf there turns out not to be an AI crash, you get a  1/(1+7) * $25,000 = $3,125If there is an AI crash, you transfer $25k to me.If you believe that AI is going to keep getting more capable, pushing rapid user growth and work automation across sectors, this is near free money. But to be honest,...

Published on April 8, 2025 4:10 AM GMTThe field of psychology is coevolving with AI and people are increasingly using LLMs for therapy. I tried using the LLM Claude as a therapist for the first time a couple weeks ago. Now I consult it daily. Human therapists likely don’t have much time until they’re partly, or fully, replaced by artificial therapists.Traditional therapy emerged in the late...

Published on April 7, 2025 10:28 PM GMTStatus notes: I take the view that rational dialogue should work with good faith people who aren't also following rational dialogue. From that point of view, this piece is about rationality. If you don't take that view then OK fine, it's about coordination.Substack versionI want to help people respect people they disagree with. In this post I...

Published on April 7, 2025 9:50 PM GMTThis post makes a simple point, so it will be short. I am happy to discuss more in the comments, and based on this write a longer post later. Much prior work (eg: [1]) has shown that exponential data and compute is required for each unit improvement in accuracy. A popular argument this leads to: Scaling compute and...