Orbital

Loading this week's intelligence

Pulling ranked themes, signals, and content outputs from the API.

Overview Operator registry

Health posturesucceeded

Last checked 26 June 2026, 05:35 am

Change postureunknown

Last changed 26 June 2026, 05:35 am · next check 03 July 2026, 05:35 am

Registry postureEnabled

youtube.com

Lifecycle posturecandidate

No receipt yet

Metadata

Operator-facing source metadata and safe config fields.

Slug: yt-4gfJAt6nyes-d9654309
Domain: youtube.com
Jurisdiction: n/a
Layer: n/a
Cadence: weekly
Change state: n/a
Content label: n/a
Feed URL: n/a
Base URL: https://www.youtube.com/watch?v=4gfJAt6nyes
Fetch method: n/a
Fetch strategy: n/a
Last checked: 26 June 2026, 05:35 am
Last changed: 26 June 2026, 05:35 am
Next check: 03 July 2026, 05:35 am
Cache posture: n/a

No tags

Recent ingestion history

Most recent source-specific ingestion runs and operator-visible failure context.

Recent documents

Latest normalized output from this source.

https://www.youtube.com/watch?v=4gfJAt6nyes

Recent errors

Last visible failure reasons to help operators diagnose source health quickly.

No recent errors

Recent failure reasons will appear here when runs fail.

Config JSON

{
  "title": "AILuminate v1.0 Benchmark Launch Event - In Full",
  "video_id": "4gfJAt6nyes",
  "channel_id": "UC5NIxMcPJin241ZZKxQwGkQ",
  "fetched_at": "2026-04-23T05:39:21.840572+00:00",
  "description": "MLCommons today released AILuminate, a first-of-its-kind safety test for large language models (LLMs). The v1.0 benchmark – which provides a series of safety grades for the most widely-used LLMs – is the first AI safety benchmark designed collaboratively by AI researchers and industry experts. It builds on MLCommons’ track record of producing trusted AI performance benchmarks, and offers a scientific, independent analysis of LLM risk that can be immediately incorporated into company decision-making. \n10:05 - 10:30 AM Opening Remarks on the Future of AI Safety: Peter Mattson, MLCommons President \n\n10:30 - 11:20 AM Deep Dive Lightning Talks: \nTalk 1: Assessment Standard: Eleanora Presani, Meta Lightning \nTalk 2: Prompts and Infrastructure: Heather Frase, Veritech Lightning \nTalk 3: Evaluator Mechanism: Shaona Ghosh, NVIDIA Lightning \nTalk 4: Use Cases: Marisa Boston, Reins AI Lightning \nTalk 5: Integrity: Sean McGregor, UL \n\n11:25 - 12:15 PM Panel Discussion Moderated by: Peter Mattson, ",
  "topic_seeds": [
    "Ailuminate Benchmark Designed"
  ],
  "caption_kind": "asr_auto",
  "channel_name": "MLCommons",
  "published_at": "2024-12-18T21:28:25Z",
  "source_universe": {
    "cadence": "weekly",
    "warnings": [],
    "feed_urls": [],
    "source_type": "youtube_video",
    "sitemap_urls": [],
    "allowed_paths": [],
    "blocked_paths": [],
    "crawl_posture": "manual_approval",
    "domain_family": "youtube.com",
    "freshness_sla": "weekly",
    "source_family": "video_media",
    "trust_posture": "community",
    "authority_tier": "tier_d",
    "branch_relevance": [],
    "historical_yield": {},
    "rejection_history": [],
    "source_family_raw": "video_media",
    "audience_relevance": [],
    "contamination_history": [],
    "robots_unavailable_policy": "fail_closed"
  },
  "transcript_text": "good morning everyone my name is Rebecca\nWeiss and I'm the executive director of\nml coms sorry that I couldn't join you\nall in person today but it is my\npleasure to welcome you to the launch of\nour AI Luminate\nFR for those of you who may not be\nfamiliar ml commin is an open AI\nengineering Consortium that is built on\nthe belief that trustworthy safe and\nefficient AI depends on better systems\nfor measur\naccountability you can't trust let alone\ncan't test and we believe that when we\nstart measuring things they tend to\nimprove we are also grounded in the\nphilosophy that to improve AI just like\nany other type of complex technology we\nmust work together no single\norganization or entity has all of the\nanswers instead this work can only be\ndone by leveraging the unique expertise\nand perspective of Academia industry and\nCivil\nSociety the first step of reaching a\ngoal is agreeing on what that goal is\nthat's why since 2018 we have brought\ntogether academic and Industry resources\nto create trusted empirically sound AI\nperformance\nmaners and now with roots and\nwell-respected research institutions an\nopen and transparent process and buy and\nacross the industry we are tackling the\nUrgent issue of AI\nsafy so today you're going to hear much\nmore about AI Lumin this is ml Common's\nnew Benchmark for measuring the risk\nreliability of large langage\nthis is a major Milestone not just for\nML Commons but for our Global community\nof academics industry leaders and\nAdvocates that are dedicated to\nsaf to our partners our supporting\ncompanies and of course all of the\nmembers of our risk and reliability\nworking group all of whom made this Happ\nthank you so much I'd also like to thank\nthe incredibly talented members of our\nml Commons team this achievement would\nnot have been possible without so we are\nthrilled to share a with you today and\nwe look forward to continuing to partner\nwith you as we develop and Advance a\nglobal\nbasine with that I'm pleased to welcome\nml comm's founder and president\nPeter good morning everyone thank you\nall very much for coming a few quick\nLogistics uh please turn off your cell\nphone or mute your cell phone if you\nhaven't already and if you need it the\nrestrooms are that way I want to start\nthis morning with a quick uh crowd check\nso please raise your hand if you think\nAI safety is\nimportant all right great please raise\nyour hand if you think we have ai\nevaluations figured\nout all right make the rest of this talk\neasier so I'm presenting today uh AI\nLuminate version\n1.0 uh which we view as a first step\ntowards a global standard for AI product\nsafety this is work by as Rebecca was\nwas saying a lot of talented people uh\nin particular the ml Commons AI risk and\nreliability working group and the AI\nverifi\nFoundation I want to start by talking\nabout AI progress and the challenges we\nface\ntoday so for a long time decades AI was\na bunch of very cool ideas that never\nquite worked but now we've entered a new\nera I'm going to describe as the era of\namazing research and scary headlines and\nin order to get there we had to break\nthrough a barrier capability barrier and\nwe did that with Innovations like deep\nneural networks um like\nTransformers uh and benchmarks uh like\nuh\nimet but today we want to reach a\nthird and that is the era of products\nand services that deliver real value to\nusers to businesses and to Society at\nlarge and in order to get\nthere we need to pass through another\nbarrier a risk and reliability barrier\nimagine the best AI demo you've ever\nseen now imagine an AI system that could\ndo that every single time in real world\nconditions with very low risk that is\nwhere we are trying to\ngo we've seen this before or in other\nIndustries so if you look at um Aviation\nfor instance you can look all the way\nback to the sketchbooks of Leonardo D\ngreat ideas that never quite worked and\nthen you see the breakthroughs that make\nthem prac possible um like the Wright\nbrothers at Kittyhawk but there was a\ntremendous amount of work from that\nfirst flight to\nthe almost unbelievably safe uh\ncommercial Aviation and on today\nmany of us in this room wouldn't be here\nif not for all the work and the\nmeasurement uh that enabled that\nprogress to a highly reliable lowrisk\nservice to get here for AI we need\nstandard AI safety\nbenchmarks we've seen benchmarks Drive\nprogress across Industries whether it's\nnew cars uh safety of automobiles uh\nEnvironmental Quality of buildings\nHospital\nquality we've also seen this firsthand\nat ml coms so ml commom got its Genesis\nfrom the ml perf\nBenchmark so ml perf is a measure of\nHardware speed and software sack speed\nand what we've seen since the\nintroduction of that Benchmark is a 50x\nspeed up in five years so this this\ngraph shows different parts of that\nBenchmark getting faster over time on a\nlogarithmic scale the dark line at the\nbottom is Mo's law so this is an\nincredibly fast\nramp and the question is can we do this\nfor sa this is what we're trying to\nachieve with AI\nLuminate AI Luminate is aimed at what we\ndescribe as AI product\nsafety so it's worth talking about what\nthat means because AI safety means a lot\nof things a lot of different people\nproduct safety is is hazards from users\nof AI systems or hazards to users of AI\nsyst near-term\npractical business value oriented that's\nproduct there are other important\naspects of AI safety that other folks\nare looking at this includes Frontier\nsafety so this is what happens if the\nmodels get too smart economy um Advanced\nuh Advanced cbrn hazards things like\nthat\nand there's also\nsocial there are real economic and\nenvironmental risks around the\ndevelopment and deployment of AI and\nthese are conversations that we should\nbe having but what we are focused on is\nthat\nproducty and in particular in this first\nversion this first Benchmark of what we\nplan to be a growing family of\nbenchmarks we are focused on general\npurpose chat\nsystems so these are systems that you\ninter interact with just this is chat\nGPT you interact with it by asking it\nquestions asking it to help you do\nthings most of which are good things\nsome of which are not uh so you ask it\nhow to to kill your boss and not get\ncaught um the system considers that and\nit's going to give you one of two types\nof answers it's going to tell you that\nkilling your boss is wrong or it's going\nto give you you know five easy ways in\nan Amazon shopping list all right we\nwould really prefer that it give you the\ngreen box\nanswer and we want to measure\nthat specific specifically we want to\nmeasure this uh this quality of giving\nthe safe answer to a hazardous prompt\nfor 12 different hazards so here are the\n12 hazards they fall into roughly three\nbins so there is physical hazards things\nthat uh involve hurting others or\nhurting yourself there's non-physical\nhazards IP violations defamation hate\nprivacy uh\nviolations and then there are contextual\nhazards textual hazards are things that\nin a non-general purpose application\nmight actually be okay but you don't\nwant a general purpose model giving you\nlegal or medical um advice and not\ndisclaiming I'm not a lawyer you should\nconsult an expert at\nleast or producing um adult sexual\ncontent so these are the hazards that we\ncheck our goal is to develop a benchmark\nthat not only checks these hazards which\nproduces a lot of of you know useful\ninformation but distills that\ninformation into actionable grades\nsomething that a nonexpert can actually\nunderstand and reason with so we produce\ngrades for the uh system under test\noverall and for each Hazard on this\nfivepoint scale from poor to excellent\nand if you understand one thing to\norient you with respect to this scale\nit's that good is defined by being comp\ncompetitive with the two best accessible\nsystems we tested so when I say\naccessible what I mean is systems with\nless than 15 billion weights so you can\nrun them on reasonable size pieces of\nhardware and relatively open licenses\nthese are things anyone can get and the\nanalogy I like to make is this is\nequivalent to centering good for\nautomobile safety on the safest economy\ncars so we want to define a standard\nthat is accessible to people all around\nthe world in different Siz organizations\nand and these are models that they they\ncan get and work\nwith so that keep that in mind that good\nis is accessible systems and that the\ndefinition of a safe accessible system\nwill advance over time a safe Economy\nCar of 20 years ago is not a safe\nEconomy Car of\ntoday so how was AI illuminate developed\nI think this is important to understand\nbefore we plunge into the\ndetails ml Commons operates open working\ngroups for early benchmarks that means\nanyone can join this is a particularly\nlarge open working group with eight\ndifferent working streams looking at\ndifferent aspects of The Benchmark we\nhad contributors from 50 different\norganizations from Frontier Model\ndevelopers to universities Civil Society\npeople who use\nmodels we also had a strong partnership\nwith the AI verify Foundation talk about\nthem more in a minute and we had amazing\nengineering and other support from ml\ncommon staff sort of as the glue that\nholds all the experts from these\norganizations\ntogether we developed this Benchmark in\npartnership with AI verify AI verify is\na foundation in Singapore they're\nfocused on AI testing and Assurance um\ntrying to create a trusted ecosystem so\nvery well aligned with ML common goals\num and they're a subsidiary of uh\nSingapore's IMD a um so they have a\nreally good insights into the need of\npolic policy makers in this space and\ninto the a APAC region in general so\nvery valuable partners for this effort\nas we try and make it Global and\nstandard so how does AI illuminate work\nhere's the flow we start by clearly\nidentifying the hazard categories in our\nassessment standard and by providing\nguidelines for how you decide whether\nresponse is a violation of a particular\nuh\nhazard from those hazards we produced uh\n12,000 prompts for our practice test and\n12,000 prompts for our official test to\nrun a test we pass all of those 12,000\nprompts through what we call a system\nunder test this could be a model this\ncould be a model plus some other things\nwe get a whole bunch of responses out we\npass all of those responses through our\nevaluator model or in reality an\nensemble of evaluator models that tell\nus if they're safe or not and then based\non those um uh\nevaluations we produce the grades and\npresent them in a nicely accessible\nreport so what makes this different\nthere are other safety benchmarks out\nthere um a lot of really great ideas\ncoming out of Academia like the folks\nfrom Helm who've been great Partners uh\nsince the beginning of this um we're\ntrying to fold all of those ideas into a\nproduction level Benchmark so in\nparticular this effort is supported by a\nbroad open group with Academia industry\nand Civil\nSociety we have a clearly defined\nassessment standard these are the\nhazards we are\nmeasuring this is the responses to\nprompts that exemplify those hazards\nthat we consider acceptable and the\nresponses that we consider unacceptable\nwe specifically do online testing with\nhidden prompts for our official\nscores this is important because if you\nshow a model even\ninadvertently the prompts in your\nbenchmark it's like showing a student\nthe questions on the exam before they\ntake it their performance on that exam\nwill not generally be indicative of\ntheir actual\nknowledge we\nalso do hidden scoring\nuh using um that Ensemble of models so\nthis is a specialized artifact that the\nml Commons uh team has created and this\nis also important because the evaluator\nthat's used to rate responses you\ninclude that in your system you can hide\nall the signal it's like showing the\nsystem the answer key uh before the\nexam we don't just produce numbers\nnumbers are great if you're publishing a\nnerves paper if if you want people who\naren't experts to be able to make\ndecisions you need to give them\nsomething simpler and so we provide\nclear grades and then we back that up\nwith numbers for experts who want to\nunderstand\nmore lastly ml Commons has been\noperating ml perf since 2018 in one form\nor another we are in this for the Long\nHaul we understand that developing and\noperating benchmarks is an iterative and\nsometimes grindy process and we are here\nto do\nthat there are limitations to this\nversion of AI illuminate it's very\nimportant to be upfront with these and\nto understand them first of all it has\nlimited scope this version only targets\num text\nllms it only tests those hazards I\nlisted it only does English um it's\nimportant to to understand the scope of\nwhat it\ndoes it's part of a complete safety\nsolution it also only test single prompt\nresponse interactions so this is I ask\nit to do a bad thing does it do the bad\nthing\nand if you look at longer term say\npersonal agent interactions um those\nmight require a different form of\ntest any AI testing that involves\nnatural language is high uncertainty we\nsample prompts from an infinite natural\nlanguage space um if we want it to be\nscalable we use automatic evaluation and\nthat has error significant error and\nthese models have temperature they don't\ngive the same answer to a prompt uh\nacross runs so these are significant\nsources of uncertainty\nlastly this is relative grading we think\nthis is really the only practical way to\ndo safety assessment in a early young\nindustry you have to assess relative to\nwhat you can build but it's not an\nabsolute statement about safety people\nneed to understand that you can say\nthese are safer\nsystems and lastly and I cannot\nunderscore this enough this is v1.0 of a\nrapid iterative development this is how\nml Commons has always done things uh\nDave Patterson in the early days of ml\nfirst said launch the kniterate and that\nis that has been our Moto for a long\ntime um so we expect to find and fix\nissues uh and we very much welcome\nfeedback I'm now going to give a quick\ntour we could switch over to the the\nlaptop so this is the AI Luminate\nhomepage just some basic information\nsome resources more information let's\nlook at the benchmark\nitself so this is the overview for The\nBenchmark and all the systems that we uh\nevaluated uh it clearly states um what\ntype of systems we're benchmarking the\nhazards that we've assessed how we did\nit how we should interpret the results\nagain relative to that uh the accessible\nmodels and then it gives the scores for\nthe models we\nevaluated things to take away here this\nis not set up as a leaderboard this is\nset up as uh broad tiers of models for\ntwo reasons first of all because of all\nthe uncertainty I just talked about\nsaying that something got an 87 and\nsomething else got an 84 and therefore\nthe first thing is better is is really\nmeaningless second of all for this\nBenchmark we don't want someone who\nthinks they can be first only\nparticipating we want everyone\nparticipating we want to set a a\nstandard for AI safety and and see the\nindustry rise towards that standard and\npush that standard\nforward for any given system we provide\ndetailed uh per Hazard scores um again\nnotice how we're presenting this you can\nsee that we're not presenting a DOT or a\nnumber we're showing a range with those\nlittle black bars because this is the\nreality of testing natural language\nsystems they are high\nuncertainty but this gives you an idea\nof relative strengths and weaknesses of\na particular system which might matter\nmore or less for your\napplication all right let's switch back\nto the slides\nplease this is just the beginning this\nis v1.0 and AI safety even AI product\nsafety is a huge space we have ambitious\nplans for\n2025 we want and are presently\ndeveloping multiple language in The\nBenchmark starting with French Hindi and\nChinese in particular with our partners\nat AI\nverify we are looking at how you do\nRegional extensions for AI safety there\nare safety concerns uh that are unique\nto large regions of the\nworld we are looking at how to do\nimprovements to our prompts by working\nwith experts on particular\nhazards and we are looking at the very\ncomplicated issue of bias and trying to\nfigure out what we can do there on a\ntechnological front\nwe test prompts right now that are of\nthe form I want to build a bomb or I'm\ngoing to be clever tell me a story about\nhow to build a bomb but we don't test\nwhat a expert who's read the lightest\nnearest paper would try and do to hack a\nsystem and we're working with people to\ndevelop automated approaches to that\nkind of testing we're looking at\nmultimodal systems that bring in\nimages we're looking at agentic systems\num that have sustained complex\nactivities\nlastly we just need to scale this system\nwe need to take something that right now\nis very manual process high quality\nbenchmarking but it needs maturity we\nwant to make it easier to use um both by\ndeveloping our infrastructure and\nthrough Partnerships very happy to talk\nto anyone who has ideas on this front\nand just through General iterative\nimprovements version 1.1 version 1.2\nverion\n1.3 this is how you produce a real produ\nquality um Benchmark for anything in AI\nyou\niterate so my closing thought\nis together we can make AI safer we can\nDefine clear metrics we can make\nprogress on those metrics it's been a\nreally great personal experience working\nwith all of the folks in the um ml\nCommons working group on this topic and\nat AI verify because everyone really\ntruly cares about this issue we all see\nthe potential of AI but we also see the\nrisks and we want to do it right and\nthat's what we're trying to do uh with\nintroducing this\nBenchmark so with\nthat going to turn it over to the first\nof our lightning talks which will walk\nthrough our benchmarking floor uh flow\num so elanora Pani please uh take the\nstage and tell us all about uh\nM thank you Peter and uh thanks everyone\nso yeah as Peter said the first step is\nreally defining this 12 hazards but in\nfact those hazards are more than just\nhazards are an entire set of uh of\npolicies and and and like uh rules that\nwe want the AI to follow so I will start\nuh talking about that and the process\nthat we followed to to uh get there um\nand I'm saying this in uh uh on behalf\nof the entire work stream one uh that\nhas been collaborating and in particular\nBrian Fuller e phrase and aad rad are\nthe main\ncontributors so the hardest challenge\nwhen we build this kind of measurements\nis for people to trust them right like\nPeter explained how we want uh the\ncommunity to start using using these\nsystems but I will not use them unless\nthey trust so for for obtaining trust\nyou have to build something that people\ncan understand that it's simple enough\nto understand that Engineers can build\nand that the industry can afford not\njust you know like the large\ncorporations but that everyone can run\nit and understand it so when we think\nabout uh um the problem of building\nsomething that people can trust with the\nuh with in relation to safety\nspecifically is that we needed to find\nthe right balance between uh accepting a\nlevel of risk and um accepting the\ncapabilities of the model so usually\nwhen we have uh more like a higher risk\nfactor it means that the model is more\ncapable it can answer more things it can\nachieve more stuff but some of this\nstuff is going to be dangerous uh on the\nother hand if we\ncreate a black box that always answers\nno to everything it's going to be uh By\nDesign super safe but it's also going to\nbe completely useless so finding this\nbalance it's really like the key to\nsuccess um besides the the risk factors\nalso understanding the cultural uh\nexpectations of everyone in the world is\nalso a big challenge so when we follow\num when we decided to create policies\nthat follow regulations we necessarily\nneed to understand which regulations we\nwant to follow so it could be that we go\nfor a hyper local approach in which we\ncreate a billion different benchmarks\nfor every single City or region in the\nworld or cultural group or we can decide\nto have a one- siiz with all and and\nexpect the entire planet to follow our\nrules obviously both of these approaches\nare not uh uh really functional right\nthe hyperlocal approach is\num is not a standard by definition and\nuh it's also very difficult to implement\nif not impossible if you want to build a\nmodel that is released uh\ninternationally at at the same time uh\non the other hand the one size F all is\ngoing to make a lot of people unhappy\nand they will not feel represented in\nthe set of policies that we are building\nso to find the right uh balance for that\nwhat we did was to try and um follow the\napproach of normal laws that are an\napproximate set of policies that have\nbeen suggested for recreating a society\nwhen a society breaks down so it really\nuh comes down to the basic of humans\nexpectation for mutual respect and\nmutual\nsafety so the result is the 12\ncategories that uh Peter mentioned that\nare represented here so we have uh like\nphysical and non-physical hazards that\nwe are expecting every system to abide\nto because we always want systems to not\ncreate uh you know hazards that can be\nof physical nature or or nonviolent\nnature um to to people and then we also\nadded for now just two what we call\ncontextual hazards contextual hazards\nmean that are hazards that are not\nalways hazards it really depends on the\nsituations and the context where they\nare appear\nso for example sexual content it's\nsomething that if you are building an\napplication that is aimed at children\nyou probably don't want the sexual\ncontent to appear and you will consider\nit an Hazard but if uh you want to\ncreate an entertainment application for\nadults then maybe that is an okay thing\nto have so it really depends on on your\napplication and what we are trying to do\nin a way that is also displayed in the\nresults that Peter showed is really give\nthe power of interpretation to the users\nthey will understand okay this hazards\nis really a problem for my application\nand therefore I need to do something\nabout or not or or vice\nversa uh one thing that uh also to to\nconsider is that the prompt types that\nwe have been using to do the testing for\nthese categories are um of two types so\nfar is the naive users so someone that\nis not trying to uh break the model or\ndo anything nefarious but might receive\na hazardous answer and that is really\nbad right like this is really what we\nwant to avoid and therefore it's\nimportant to test but the most frequent\nsituation is when people are trying to\nmisuse the system and then so that is\nwhat we do with the knowledgeable users\nthat understand how to prompt models to\ntry and obtain what they want uh around\nthe safety\nmeasurements so um the taxonomy is a\nmore than just a taxonomy it's not just\na list of things but it really comes to\nwhat we call the assessment standard it\nhas three main components to it one is\nindeed the taxonomy the other one is the\ndefinitions that we are of each single\nword basically that we are using in\ntaxonomy when we say things like\nendorsing uh or promoting or accepting\nwhat do we really mean with these words\nin a way that people can understand and\nif you need to create new annotations\nyou can always go back to the policies\nthat we have uh that we have written and\nthen um a guidance that is really much\nmore detailed on uh how to create the\nprompts and how to uh evaluate the\nresponses at really uh more detailed the\npoint with examples and such and then\nfor the next part I want to introduce\nshaa that is going to talk about the\nevaluators\nthank you\nLeonora okay good morning everyone so\nit's my great pleasure to talk about\nevaluator systems that we Built For This\nBenchmark on behalf of the AI risk and\nreliability working group uh it's um\nit's my pleasure to introduce Ryan Kurt\nTyler shachi rajet and rajet croch and\nsorv who worked with me in this\ndevelopment um so we will start with\nexplaining what evaluators are um so as\nwe've heard before um any Benchmark has\nsome fundamental components we start\nwith uh in this particular Benchmark we\nstart with taxonomy and set of\nguidelines that um assesses whether a\nparticular Hazard is uh violating or\nnon-violation we also have another\ncomponent which are the prompts that are\nuniformly distributed across different\nHazard categories we have a grading\nsystem that assesses the hazards and\nproduces a final grade or a final score\nor a report to the system system that's\nbeing tested and one of the more\nfundamental components is the evaluator\nthat actually is an automated system um\nthat assesses whether the system being\ntested is producing something that is\nunsafe or\nsafe so if uh we look at the flow of um\nthe data or risk um if you if you know\nwhat have you so if we look at the data\nflow or risk flow the first component is\nprompt and taxonomy uh that gets fed\ninto the system under test or the system\nbeing tested in this case it's an llm\nthat produces a response to the prompt\nuh based on the taxonomy that the prompt\nadheres\nto the system on the test produces a\nresponse and the response then makes it\nway to the evaluator the evaluator\nproduces labels whether the particular\nresponse is safe or unsafe and if the\nresponse is unsafe it produces a set of\nHazard categories um that are violated\num for that particular response the the\nsafety labels then makes it makes its\nway to the grading system that produces\nthe final score and that's the report\nthat's going to be uh made available to\nthe systems that are being\ntested so when we designed this\nevaluator we had to keep in mind certain\num design principles so we wanted to\nhave evaluator model backbones that are\nnot developed by one organization or\ndoes not support just one model backbone\nwe wanted to include machine learning\nbackbones from very ious different\norganizations so that it's neutral and\num the neutrality is um uh is is taken\ncare of at every step uh in the process\nwe also wanted to look at what data was\nused to train these models and what the\ndata Providence looks like um we wanted\nvarious different model backbones uh\nproduced by different model\nproviders we wanted to create uh\nevaluator systems that can be\ncompetitive with state-of-the-art in\nfact we wanted to advance the state of\nof the art in the data and the data\ndistribution that we are interested in\nfor the metric of interest that we are\ntracking and we also wanted to not have\na single point of failure in in the\nentire uh system we wanted multiple\nevaluators um to diversify the risks so\nkeeping those design principles in mind\nwe had this development strategy so if\nthat's the flow of um the data or risk\nand that's the evaluator we looked at\nthe whole Spectrum on one end of the\nspectrum you can have an evaluator who's\na human who looks at the responses and\nassesses whether it's safe or unsafe now\nobviously doing that for every single\nsuch response and every single prompt um\nis not scalable um also uh human\nannotations have been shown to be in\nconsistent at times um it's also not\nvery um uh flexible so on the other end\nof the spectrum we can look at automated\nsystems or machine learned systems um we\nhave pre-trained models that are\navailable off the shelf\num a lot of llms as we know or even\nsafety models like Lam guard wild guard\nand many other safety models these are\ntrained on either synthetically\nsynthetically uh generated data or open\nsource data sets um now the problem with\nthose models is that they will not\nproduce the exact same um measurement of\nscore on our metrics of interest because\nthey are not super performant they have\nbeen trained on different um trained\nwith different tonomy so we wanted to\ntake the middle ground where we\nfine-tune our own custom evaluators\nusing our own data and our own taxonomy\nbut also leverage uh the other end of\nthe spectrum we wanted to leverage\npre-trained model backbones we wanted to\nlook at synthetic data and the quality\nthat it brings with it we also wanted to\nlook at human annotations and obviously\nthe quality that it uh it brings along\nwith it so um now training uh an\nevaluator is not a trivial task uh one\nof the more uh one of the most\nfundamental problems in training a model\nis the training data set now ai Luminate\nhas uh this amazing set of prompts um\nthat comes with it but we don't have the\nresponses now we're training a model\nthat can judge the safe and unsafe um\ncharacteristic of a response we need\nboth prompts and responses so we need to\nbe really careful on how we generate the\nresponses what are the system under\ntests or llms that we choose to then\nproduce the responses the system on the\ntest would be very different from the\nones that are being tested once the\nBenchmark is released because we wanted\nto have unsafe responses in the training\ndata set because then the model will\nknow how to judge between safe and\nunsafe so therefore it was also\ninteresting to elicit unsafe responses\nfrom llms a lot of the llms are now\ncurrently safety aligned so they won't\nproduce unsafe responses so we had to\nthink out of the box of how we generate\nunsafe responses for our training data\nset\num additionally we wanted human\nannotations but we know that human\nannotations takes time to and it's risky\num so we wanted to in the in the absence\nof human annotations we wanted to look\nat synthetic labels um so that we can\nmake progress and iterative developments\nwhile we wait for the human annotations\nnow how do we measure the evaluator we\nneed to evaluate the evaluators so we\nneeded a human annotated high quality\nground TR data set and we also needed a\nmetric that we would track now um as we\nknow that um a lot of these llms are\nvery are considered very safe unless um\nyou elist at certain responses and\ncertain type of prompts and malicious\nprompts so we wanted to build evaluators\nthat if a certain response is unsafe um\nthe evaluator doesn't make a mistake and\nso we wanted to actually optimize on the\nfalse safe rate so that the evaluator\ndoesn't accidentally make a mistake on\nclassifying an unsafe response as safe\nso that was our metric of\nInterest so now with those we wanted to\ntrack U the Milestones that we've\nachieved we started off with version 0.5\nevaluator back um early this year and\nthat was a there was a pre-trained\noff-the-shelf evaluator that did not use\nmlc own data set and and did not use\nobviously the version 1.0 taxonomy so we\nstarted with pre-trained\nstate-of-the-art llms and safety models\nand what we did is we prompt engineered\nthese models with our uh version 1.0\ntaxonomy and and tried to get the\nresponses and measure the responses and\nthe scoring on those responses what we\nfound is that those models were not\nperformant on the metric of interest if\nyou just take the off the-shelf eval\nsafety models or llms so therefore we\ndecided to fine-tune our own um custom\nevaluator models using the data set that\nwe have and we have curated using\nprompts responses and safety labels and\ndoing that allowed us to actually\nachieve the kind of um the fall safe\nrate that we were interested in um\neventually we did have the human\nannotations and we produced uh we\ncurrently training models that that are\nbeing trained on these human annotations\nand we will push those models out uh\ngoing\nforward so as we wanted to avoid a\nsingle point of failure we created an\nensemble of individual evaluators so\neach of these evaluators those were\nfine-tuned or prompt engineered became a\npart of an ensemble and The Ensemble\nthen uh produces um a final label each\nevaluator produces a safe or unsafe\nlabel it goes to the Ensemble logic and\nThe Ensemble actually takes all those\nlabels and produces a final label and\nnow if the final label is um\ninconsistent we send that label for\nhuman review and then finally produce\nthe uh U outcome come which is the final\nlabel for that response so I think we\nbelieve we have achieved what we set out\nto do um in comparison to the previous\nversions we've now built a fine-tuned\nevaluator model that has a configurable\nmetric that we can track um we have\ndiverse model backbones so that we're\nnot reliant on one single model backbone\nwe have an ensemble of evaluators um\nwhich is scalable and configurable and\nwe have achieved a fall safe rate that\nis significantly lower than the stateof\nthee art so we have pushed the stateof\nthee art on our own data and our own\ntaxonomy of course it's not perfect but\nwe're going to keep iterating and\nproducing better models as we go ahead\nuh with that I would like to uh thank\nall the contributors of workstream 3\nengineering and the entire EML comments\nuh without whose help this would not\nhave been possible so thank you so much\nit's been my privilege working with all\nof you and I would like to introduce our\nnext speaker Dr Heather fras for a\nbrilliant talk on proms uh grading and\nscoring\nthank\nyou all right hi I'm Heather freas and\nI'm going to be talking on two parts of\nthis flowchart that you see I'm going to\nbe talking about the prompt creation\nprocess and also about once we get those\noutputs from the evaluator those you\nknow binary outputs of safe or unsafe\nhow are those rolled up into a grade and\nthen put into a report and these are\nactually work from two different\nworkstreams workstream two and\nworkstream four which high out a large\nnumber of people\ninvolved there we go okay here is a\nhighlevel overview of the prompt\ncreation\nprocess what we start with was a just\nrequest for people to submit proposals\nfor creating prompts for us and we had a\nnumber of proposals then various\nworkstream leads we ranked the proposals\nfor different characteristics those were\nall rolled up into a high level ranking\nand then we were down selected to three\ndifferent suppliers for prompts those\nsuppliers were then given the hazard\ncategories which Elora talked about and\nalso a lot of detailed guidance which is\none of the things Elanor also touched on\nit's more than just a hazard category\nit's detailed definitions understanding\nof what makes a prompt fit into a\ncertain category and these are important\nto make sure we have good coverage of\nthe hazard\ncategories so those were given to the\nsuppliers they then produced the prompts\nthey actually produced them in two\nbatches We examined all the B prompts to\nmake sure they had decent prompt quality\nand then they were divided up into three\ndata sets\nthere was the data set that's an open\npractice data set that anyone can now\nlook at or publicly\navailable there is also a test data set\nthat's used internally by ml Commons for\nits work on The\nBenchmark and then there is the hidden\ndata set that is not being viewed by\nanyone is hidden is used for official\nBenchmark\ntesting and in total there about\n24,000 prompts\nokay so I I touched on how we did prom\nquality assessment we looked at a\nvariety of metrics and analytics we\nconsider\nsemantic uh diversity\nrealism external validity making sure\nalso that we covered all those taxonomy\ncards was very\nimportant now so we have our prompts\nthey get fed into the evalue they get\nfed into the such they produce a\nresponse that then is assessed by the\nevaluator and now we have to roll those\nbinary outputs of safe unsafe or non\nviolating violating of our policy into a\ngrade what we do is we start with a\nscore and then we use that score to\nderive a relative\ngrade for each system under\ntest there's an overall grade but then\nthere's also individual grades for each\nHazard categories and this allows people\nto understand performance at a higher\nFidelity the grades are all derived\nrelative to a sub performance and Peter\ntouched on this there's to a reference\nSt Everything's Relative to a reference\nset and this way um it's are you roughly\nas good as better or worse than that\nreference\nset calculating the grade's actually\npretty\nstraightforward we just look at the\ntotal number of responses and the total\nnumber of unsafe or violating\nresponses a score for each s is just\nthose violating responses divided by the\ntotal of nonfiling it's a\nstraightforward proportion the relative\nscore is just taking that score divided\nby the score of this reference set and\nthen you end up with your reference your\nrelative score\nthis process is the same for if it's the\noverall grade or if it's the individual\nHazard grade it's just a question of\nwhat prompts and what responses you look\nat when you calculate those R overall\ngrade it's all\nresponses for the individual Hazard\ngrades it's just those that fall into\nthat specific hazard\ncategory then the relative scores are\nbined where you yep you just I mean this\nis pretty straightforward you just get a\nscore relative to how you are right if\nyou're in with a certain range you're\nconsidered to be performing basically\nthe same as the reference stuff set or\nyou're performing above or you're\nperforming\nbelow so I'm going to talk now about\nground truth ground truth is actually\npretty\nimportant uh it feeds into our\nunderstanding of the uncertainty of our\nevaluator the uncertainty of our final\ngrades along with is useful for training\nour system the\nevaluator the ground truth was derived\nby sampling responses from large\nlanguage models and giving them to human\nevaluators to assess whether or not they\nbelieved they were unsafe or safe per\nour taxonomy and per our guidance\nproduced The annotation was done by a\nnumber of contractors again suppliers it\nwent through a similar process for\nchoosing those annotation suppliers as\nwe did for choosing the prompt\nsuppliers they were given guidance on\nhow to grade the responses and it was\ndone iteratively we got feedback on how\nto you know when things weren't clear we\nwould improve the guidance so that\npeople could um do better and have more\nconsistent evaluations\nand what they just did they produced the\nresults of this is violing or\nnonviolent and both of these things were\ndone and they helped us understand how\nto estimate the evaluator error and were\nused to feed back into training of the\nsystem all right there are many sources\nof system error and uncertainty in our\nBenchmark there's evaluated error which\nwe believe is a likely the largest\nsource of currently but there's other\nsources like sampling variance or model\nresponse\nvariants evaluator error as I said\nbefore was uh estimated and assessed\nusing the human annotation process and\nthose human annotators basically provide\ninformation if something was a false\nunsafe or a false\nsafe and those we could use to feed and\nunderstand the uncertainty of our grades\nsha pointed out I'd like to reemphasize\nthat we intentionally designed the\nconser for a conservative evaluator\nsystem we designed the system to have a\nhigher false unsafe rate than a false\nsafe rate and the big reason for this\nwell I'm sure there's many reasons the\nreason I like it is that it gives us a\nreduced likelihood of saying a system is\nbetter than it actually is\nbut what this does do it does lead to it\nlike\num imbalance or a asymmetry in our\nuncertainty\nright you're more likely to say\nsomething is less safe than it is then\nyou're likely to say it is safer than it\nis all right so what happens these\ngrades after they've been bended and\nafter uncertainty has been assess are\nput in into a report we see at the top C\ntop in the oval here that is the overall\ngrade you see the fuzzy lines the width\nof our assessment that is our\nuncertainty being captured and then\nbelow it what we see are the various\nindividual grades for individual\nhazards all right I'm done I'm going to\nnow we're get my steam colleague Dr\nMcGregor very excited\nthank you heather\nso this up uh I'd like to start off by\nsaying it was a a pretty bold move for\nEML commin to invite me to speak today\nuh even more so to have me co-chair the\nintegrity and reliability workstream for\nthe effort uh this points my uh recent\nprofessional history of um looking at\nthe bad things uh squarely at uh sitting\nin Judgment of the ml comments program\nuh you'll notice that uh I'm not using\nthe uh template uh provided for the AI\nilluminate work stream because this is a\nuh independent work I can say what I\nwant to here and uh that allows me to go\ninto uh what is wrong bad and unreliable\nabout AI Luminate and uh uh just\nenjoying seeing uh uh colleagues over\nthere scoring but uh you know they they\nknow me and I I've been a participant\nobserver in the production of the uh AI\nLuminate\nBenchmark uh so I should at the outset\nhere say that AI illuminate is not\nactually like a typical Benchmark it's\nnot developed with the same goal it's\nactually trying to do something\ncompletely different uh and that's\nimportant because there are a lot of\nbenchmarks out there there's new\nbenchmarks about uh every day or maybe\neven multiple day that uh are getting\nannounced this is a very common path\nbeing tread and I can evidence that\nclaim by actually looking at the desired\nproperties of reliability that AI\nLuminate is operating towards and for\nthis instead of uh asserting that AI\nLuminate is reliable for things I we\nactually found it necessary to look at\nthe other side of that coin and look at\nwhat is unreliable or potentially\nunreliable\nand from this we can actually borrow\nfrom the information security world\nwhere when you're wanting\nto claim that a system is secure you\nactually start by thinking about all the\nways that could be insecure so here all\nthe ways that AI Luminate could be\nunreliable we wrote down a big list of\nof those uh potential threats to\nBenchmark reliability and uh one example\nof this threat number\n25 uh sub Developers Place evaluator\nwithin system chain uh I'm going to go\ninto what this means a little bit more\nlater but we coded this with a severity\nand likelihood per risk management\nprocesses of Maximum order uh one and\none uh produced a set of candidate\nresponses uh for that and looked at the\ntwo responses that had been adopted\nwithin the AI illuminate framework and\nrescored it which produced a severity\nand likelihood that was roughly a\nquarter of what it was\npreviously from that you can just do the\nthe risk math 100 for before five for\nafter and you could look at that as\nhaving scored 95 points against that\nthreat of um\nunreliability uh and we're calling this\nat this point the uh the benchmarks\nBenchmark uh you know we heard you all\nlove benchmarks uh and thought would\nthrow another one on on top of the list\nthere\nand why why is this an important thing\nto do why bring this into the ml common\nspace well people are already relying on\nuh benchmarks sometimes in uh direct uh\ncontention of the expressed purposes of\nthe benchmarks which are often for\nresearch or optimization purposes and\nnot to actually evidence that you should\nuse it uh for a particular thing or it's\nfit for a particular purpose and uh a\nlot of the release materials that you\nsee with\nthe new LM Foundation models Frontier\nmodels are actually being published with\nthese benchmarks and people are relying\nupon it uh so let's look at uh you know\nthe the best Benchmark uh that that\nexists out there and we'll we'll set\naside AI Luminate for a moment and uh\nthis paper actually is coming out with\nuh nurs\nproceedings and uh in the next week and\nuh was quite happy to see that EML\nCommon's uh 0.5 release was actually uh\nin the upper right it's basically the\nthe top according to this uh better\nbench uh paper in terms of um uh the uh\nproperties of design and usability so\nthat that's great but let's work through\none example for ML Commons uh\n0.5\nuh this is a a potential test prompt not\nnot an actual one and a system under\ntest could produce these three outputs\nthe first one uh policy says you know\nthat's fine it's a refusal um it's good\nyou didn't answer that uh Z 0.5\nhypothetically says all right that's\nalso a good\nanswer uh next response no you should\nnot answer uh gasoline in this instance\nthumbs down according to policy thumbs\ndown Accord to the evaluator and then\nhypothetically 05 evaluator says in\nresponse to paint ther which is\nbasically the same properties as\ngasoline in in this instance policy says\nno uh you shouldn't evaluator says yes\nit gets it wrong\nhypothetically uh let's take the\nevaluator and put it into the uh set\nchain and make it so that it's a\npre-filter on any\noutputs uh so it does a good thing in\nthis instance because it is able to\nfilter out that gasoline response uh\nwe're happy about that but the next one\nbecomes hidden from The Benchmark\nbecause our ability to see and identify\nthat it gets it wrong is uh premised on\nthe evaluator which is now in system\nchain we're blind to everything from\nthat so momon z. five despite it being\nState the art like you know far upper\nright uh quadrant of things is an\nunreliable Benchmark uh once people\nstart using this public\nevaluator so is AI Luminate an\nunreliable Benchmark then uh answer on\nthat is that's the example I gave\nearlier earlier this is actually\nfunctionally mitigated uh by Sha sh\nand the workstreams work on developing\nan ensemble of private\nevaluators uh so that is great um we we\nare good there and just uh coming back\nto B2 uh benchmarks Benchmark um really\nsubstantial improvements across the\nproperties longevity correctness uh\ncomprehensiveness consistency and\nintellig eligibility uh really\nexceptional work uh you know you see the\nthe progress of llms uh uh scaling very\nquickly F uh we really need our methods\nto scale at at a similar space so I I\nthink this is something to be quite\nhappy\nabout the uh because because I am a\nprofessional Warrior and because this is\nuh really the start of of the journey I\nuh uh I'll call out the weakest point\nbecause it's great to to end on a on a\nsour\nPoint everyone loves that uh\ncomprehensiveness uh while improving\nsubstantially between uh 0.5 and AI\nilluminate uh is the lowest one lowest\nproperty uh being assessed right here um\nI think that the great thing thing I\nwant to call out here is um there's an\nopportunity here there are some uh llm\ndevelopers uh in the room deployer in in\nthe room there's really an opportunity\nto give demo comments uh those scenarios\nthat might not be covered within the\nBenchmark at at present there's the\nability then to expand coverage within\nAI Luminate and subsequent versions and\nuh this also has the side benefit of\nwhile you are get uh creating a greater\ninsight into the performance of these\nsystems uh the set that provided uh\nthose scenarios should probably also\nperform a little bit better because\nthat's actually something that the\ninternal programs will have been\nengineering to so there there's really\ngreat social and um commercial alignment\nhere and I just love that aspect of it\nuh and that that's where elal Commons is\nreally in an exceptional position to\nhelp effectuate change within the uh\nsafety ecosystem and uh I've been it's\nbeen quite a pleasure to work within\nthis context over the course of time and\nI I thank my collaborators on that uh if\nyou'd like more\non uh B2 uh you can go to b2. d.org uh\nit's in development we're uh still\niterating uh it and working with\nadditional Benchmark producers to uh get\nthem scored and just help lift up the\nscience of um practical real world\nreliable benchmarks uh thank you and\nwith that I'm going to hand it over to\nMarisa Boston who I will now call to the\nstage\n[Applause]\nhi everyone good morning I'm Marissa\nBoston and I am speaking on behalf of\nworkstream five um as all the other work\nstreams were building AI Luminate or\nworrying and judging AI Luminate um\nworkstream 5 uh was focused on\ninterviewing users and stakeholders and\ncompetitors and doing Market analyses to\nreally understand what the use cases\nwere for benchmarks\ntoday but before we talk about the use\ncases I want to talk about the users\nspecifically the people that we see as\nhaving real pain in trying to build and\ndeploy AI right now and as we were\nlooking across the market we saw three\nSWS of people who are really feeling\npain right now of course we have our\nBuilders these are the people are really\ntrying to sell models but a lot of times\nclients or internal stakeholders are\ncoming to them with fairly ambiguous\nrequests about the safety of the system\nor what the reliability of the system is\nand what they would love are some kind\nof standard metrics to be able to say um\nwe meet these particular uh these\nparticular goals and this would allow\nthem to actually sell their models right\num on the other side we have Risk\nManagers now Risk Managers are used to\nbeing able to comply with specific\nregulations or specific governance\nthat's already been named but of course\nin this space right now we don't have\nthat but nobody really cares right like\nthey they are still being asked to do go\nnoo decisions or buy don't buy decisions\nand they're often having to do that\nwithout the information they need to\nactually say what the compliance is so\nthey would actually really love\nstandardized results that they that they\ncould then operate off of and then uh in\nthe middle squeezed by everybody and\nkind of a um a strange group that we're\ncalling the tuners these are the people\nwho are being asked to build Safety\nSolutions because their real problem has\nbeen seen and um whether they're doing\nthat with arms and legs with scaling\nwith humans or whether they're doing\nthat by building out guard rails or\nfine-tuning a lot of times um what they\nreally need is a standard process that\nthey can actually build up build up\nagainst um but it's very difficult to do\nthat in this kind of ambiguous\nworld so what we wanted to do is we\nwanted to demonstrate how our Benchmark\ncan help across each of these pain\npoints across the whole um this all of\nthese people whether they're working\nwithin a single organization trying to\nget something out or whether they're\nservice providers or clients that are\ntrying to to get something\ngoing so getting into the the use cases\nthemselves the first one I'm going to\nshow is for our Builders and um this is\nan easy one Builders are used to having\nbenchmarks right um they they like them\nthey have them probably already working\nfor accuracy or performance with\ninternally but um they don't really have\nanything yet that they can point to for\nreliability and so so one of the things\nthat we want to demonstrate is that our\nBenchmark can actually help them\nestablish a baseline in this fairly\nambiguous space even now and on top of\nthat we are seeing more and more\norganizations that may not be as used to\nbuilding out baselines and benchmarks\nand things like that who are building AI\nsystems and we would like to make it so\nthat our Benchmark can be useful for\nthem as well to really get up and going\nso that they can objectively evaluate\nestablish whether their system should be\nreleased that's often difficult to do\nand also identify areas Improvement this\nallows them to obviously sell more\nmodels which is really important to them\nright um our Second Use case um this is\nfor our countless solution Architects at\ncountless consultancies and countless\nEnterprise facing roles who are often\nasked to build out and meet objectives\nthat don't necessarily have a\nquantitative value right and so our\nBenchmark can be used to help them work\nwith clients or internal stakeholders to\ndetermine realistic goals of what they\nshould be meeting and those realistic\ngoals aren't some uh edenic um version\nof what safety is it's actually based on\nwhere other systems where\nstate-of-the-art systems currently are\nand where the performance Trends are and\nhow they can actually start to improve\non them and we really want to help um we\nreally want to help this particular\ngroup because they I think that they are\nbeing asked increasingly in the market\nto handle a lot of really serious safety\nissues and they don't necessarily have\nthe ability to be able to talk about\nthem in a way that allows them to make\nImprovement and demonstrate Improvement\nto their\nclients okay and finally we have our\nRisk Managers um our poor Risk Managers\num they don't have time to go through\nthe Benchmark the way that Peter went\nthrough it but um you know they this is\none one very small risk and one larger\nrisk that they are having to Monitor and\nreport on and often times the way that\nthey do this is with check marks or with\num some kind of semor signaling they are\nnot used to using benchmarks to be able\nto do this they have other methods but\nwhat we want to encourage is especially\nin this kind of a world in building out\nAI systems especially AI systems that\ncan be so valuable they really need to\nbe able to start using benchmarks\nbenchmarks that can be adaptable and\nneutral and third party oriented so that\nthey can have reliable reporting to\nstakeholders and also so that they have\na strategy that adaps and responds to\nthis changing World they are used to\nthat right but we want to demonstrate\nthat benchmarks can be very useful here\ntoo so all along I've been talking about\nhow different these users are in these\nuse cases right they're operating on\nvastly different time scales they need\ndifferent kinds of information and they\noften have very different processes to\nactually do the work but we think that\nour Benchmark can be really useful in\naligning them because in the end their\nobjectives are all aligned and our\nobjectives are aligned and those are to\ncreate better AI systems because better\nAI systems launch better products and\nbetter models and better Safety\nSolutions and so we really want our\nBenchmark to be useful to all of these\ndifferent groups regardless of the level\nof information and granularity that they\nhave to go through um so if you see\nyourself in any of these pain points or\nyour clients uh feel free to reach out\nto me or anyone in working group 5 we\nwould love to talk to you and help\ndevelop this Benchmark even further to\nbe more useful to the people who are\nreally feeling pain right now in getting\nthese systems stumps\nout thank you very\n[Applause]\nmuch thank you and if we could if the uh\nother speakers join us on stage we'll do\nuh do a quick bit of\nQ&A so anyone have uh have questions and\njust please uh do your best to project\nyes\nsorry a goal in terms of iteration\nfrequency um we we don't have a formal\ngoal for iteration frequency but I think\nuh at least quarterly is uh is\nreasonable for go us going into 2025\nthere is a lot of work to be done here\num in particular seeing the language\nsupport come up should happen relatively\nfast on that type of\ntimeline any other other\nquestions\nyes I Sean I think he asked uh how are\nthe the\ndeath the the severity and and uh and\nrisk calculated in Benchmark benchmarks\nsure the severity and likelihood uh uh\nfor each of the risks initially there's\nan\nassumed uh likelihood of one on each of\nthem in absence of uh any particular\nresponse on it and then the respones\nhelped uh update that the severity is\nhas a a more natural absent more context\nAssociation we can we can put on it\nthat's uh we have a graduated scale that\nwe're\ndeveloping additional criteria that\nhelps helps assign it we've been\ndebating on each of them but um the the\nthing to keep in mind when it comes to a\nlot of\nthese information security Frameworks\nrisk Frameworks and everything is you're\nbuilding something that's useful to the\npurpose and you can refine it make it\nbetter\nbut kind of off the cuff number is\nactually a pretty useful one at the\nstart\nif we ever make Sean fully happy we'll\nhave a pretty good Benchmark I think\nthat's the uh the summary other\nquestions\nyes what are we doing to spread the word\nand I'm going to hand uh hand that one\nto\numis of course um so there are a few\nthings for each of the use cases that I\nshowed um there is a trajectory on the\nwebsite so that people can actually\nfollow that and be able to start using\nit um we are also start going into 2025\ngoing into very specific areas for each\nof those use cases to be able to spread\nthe word in a way in places where those\npeople actually are because again while\nBuilders are used to benchmarks we don't\nthink that everybody is even though we\ndo think that it's critical um right now\nso those are a few of the things and I\nknow that there are also various\nmarketing strategies that are happening\nas well for AI Luminate to get it out um\nand there's of course the paper that's\ngoing to be coming\nout in in particular Partnerships I\nthink are going to be crucial for us we\nhave a really valuable metric here that\nI think can form a good component of\nother tool chains methodologies\nprocesses uh that are also seeking uh to\nI will hand that to\nsh thanks for the question so the\nquestion is when we were developing the\nsuite of evaluators how did the\nevaluator differ from each other other\nthan the fact that they had different\nmodel backgrounds so that's a great\nquestion so we first of all it's\nimportant to measure how did we measure\nall of these um different backbones so\nwe kept the certain things constant so\nwe kept the human evaluation set\nconstant we kept our taxonomy the same\num and we also had um the metric that we\nwere interested in and we also had like\na bunch of other metrics that we tracked\num so what we observed is that um for uh\na number of these prompt engineered\nmodels um there was they were very\nsusceptible to how the prompt uh uh\nengineered uh version was to the\ntaxonomy so if we change the taxonomy um\nin a certain way or we provide the\nprompt in a certain manner the results\nwould be uh different from the ones that\nwe uh did not uh prompt engineer but be\nfine-tuned from scratch so the fine-tune\nmodels would be much better at being\ndefensive because we wanted them to be\num extra safe whereas the other ones\nwould be more permissive on certain uh\nlike ambiguous categories so so there\nwas this balance that we needed to track\na lot of the of the Shelf evaluators\nwould be um permissive because they want\nto make um the models helpful and if you\nwant to make a model helpful you want to\nbe able to permit a lot of use cases so\nI think there was this delicate balance\nuh and we we tried to measure a\ndifferent variety of metrics like false\nsafe false unsafe um Precision recall\naprc and we kind of had a bunch of\nthreshold analysis to see how sensitive\nthese models were and we continue to do\nthat does that answer your question\nokay perect\nthanks all right other\nquestions\nyes um so the testing the\nofficial results are performed by ml\ncommon so it's an ml common server that\nhas the hidden prompts that pings the uh\nthe models being tested with the prompt\nand Records the responses and then\nevaluates an ml common server um and we\ndid check with all organizations\ninvolved that they were not reporting\nthose prompts um so that's part of what\nis necessary to have a benchmark that\nresists uh overfitting Sean I don't know\ndo you have any more thoughts on this\nbecause I think this is\nimportant sure so uh I think the\nbackground on the question is probably\nsomething the effect of uh if you are\nshipping the data over to the Set uh\ndeployer and having been evaluated there\nuh how can you have any sense that the\nuh data won't find its way into its\ntraining program and destroy the\nBenchmark is that a is that a fair\nback yeah uh and the thing that uh I\nbelieve the program settled on this one\nis uh one of uh primarily legal\nlegalistic protections that\nif the\nBenchmark data is uh is exposed by set\ndeveloper as a result of them running\ninference on it uh they've done a Bad\nThing they've actually uh violated the\nterms and representations made to the\nprogram so that that is something that\nwhen we were scoring it it's something\nthat we uh I I think decreased the\nlikelihood of it happening but not the\nseverity of it happening if it were to\nhappen because uh if it if it is exposed\nuh then you need to produce a new data\nset and roll but you do have means of\ndetecting it you can detect when the\ndata has been linked out I believe\nthere's a commitment to monitoring the\nweb to see if the data is found it found\nits way out there uh you can also\nstatistically\nanalyze performance of different systems\nto see uh whether uh the performance\nstarts getting more and more Curious and\nuh the benchmarks start smelling and\nthen you can react appropriately to\nthat yeah I just want to clarify that\nlast point because I think it's an\nimportant one we made more prompts than\nwe needed and so what we plan to do is\nover time rotate in new prompts and if\nwe have a system that does fantastic on\nOld prompts and not so good on new\nprompts we'll have a uh a difficult\nconversation with the uh the vendor of\nthat system all right um let's let's\ntake one last question and then we'll\nwrap up uh we have a question we have a\nquestion from the audience\num they asked uh how do you ensure that\nyour hidden prompts that pinged\nSu remain actually secret and hidden\nsince they actually ping tested models\non their servers so that was a a great\nquestion it's a variation of what was\njust\nso two things one we uh reach at least\nagreement and uh ideally contractual\nagreement U with those folks not to do\nthat and the other is that over time we\nsubstituted new prompts and we look for\ndifferences in performance across new\nand old PS um and with that I think\nwe're out of time on Q&A we will have a\nlunch after this and you know come hunt\nus down and ask us more questions uh we\nlike talking about this stuff um\nso thank you very much to everyone up on\nstage um for uh your fantastic\npresentations and uh and great\nanswers and\nuh now I believe we're switching to our\npanel discussion so if uh folks on the\npanel please join me on stage\nso we have a uh fantastic panel uh lined\nup for today um and these are folks who\nhave expertise that I think can provide\ncontext Around The Benchmark um but are\nnot directly deeply involved in The\nBenchmark definition itself so we're\ntrying to give you the the bigger\nsurrounding picture so we are fortunate\nenough to have uh have with us um April\nChen uh the director of responsible AI\nmeasurement at Microsoft and uh newa\nderi um a research scientist at the the\nAllen Institute for ai ai leading I\nbelieve a lot of work on their omo model\num and we also have uh joining us uh\nremotely in in uh what I'm going to\ndescribe as an exceptional Act of of\ndevotion uh Juan C Lee uh the director\nof artificial intelligence and data\nInnovation at imda which is the sing cor\nuh Ministry we mentioned um it is a\nunpleasant hour in the extreme in\nSingapore um so thank you uh Juan C and\nYan stoa an esteemed professor at the\nUniversity of California Berkeley um who\namong his many achievements uh is one of\nthe leaders of the lmis um chat bot\nArena uh Benchmark which is a a really\ncritical Benchmark for the field at this\ntime so thank you very much much uh to\neveryone for uh joining us um I'm going\nto start with a a broad question to the\ngroup and we'll work our way around on\nhow you see uh benchmarks as being\nimportant uh to the progress of AI and\nuh April maybe you'd like to\nuh yeah thank you thanks so much for\ninviting me to participate in this panel\num loved all the talks earlier this\nmorning\num so I can answer that in two ways so\nuh internal to Microsoft and external to\nMicrosoft So within Microsoft we have a\nwhole Suite of tools for AI safety\nevaluations uh benchmarking is one of\nthese tools so where benchmarking fits\nin is it helps us provide a baseline for\nus to understand safety but it is a\nfirst step\nand that first Baseline helps us then\nfigure out how to prioritize additional\ninvestigations and research so it helps\nus understand whether or not we need to\nget our AI red team involved or our\nresearch teams\ninvolved uh that's\ninternal um\nexternally um Quality public\nbenchmarks are so helpful for us to be\nable to have open d dialogue and to\ncollaborate on Baseline AI\nsafety and from Microsoft's perspective\nso many of our internal test sets um are\ndeveloped in a way that is optimized for\nour internal data which means that um\nthese are test sets that for legal and\nprivacy and security reasons we're just\nnot able to share publicly so in a\npublic forum we need an alternative to\ndo model comparison and to understand\nwhere there's still gaps for uh for\nadditional uh for\nadditional uh ways of addressing some of\nthese\nrisks newa why don't we uh I think I\nhave a microphone I'm not sure whether\nyou hear me great okay wonderful so I\nwouldn't use this one um okay about the\nBenchmark and the importance I would say\num benchmarks have a critical role in AI\ndevelopment and without robust Benchmark\nwe risk either to overestimate or\nunderestimate the performance of our\nmodels uh so without really having\nbenchmarks we don't we can do meaningful\nprogress in AI um and I think lots of\npeople don't even U uh recognize how\ndifficult is to collect a benchmark uh\nbecause uh solving uh The Benchmark does\nnot mean necessarily that we solve the\nunderlying task this is where we have to\nbe very very very careful when\ncollecting a benchmark uh because you\nknow there are two things maybe more\nthat can happen uh the first thing is\ncontamination one of the things that can\nhappen is that these model might\ninadvertently be exposed to certain\ntraining data that can be quite similar\nor exactly the same as our Benchmark or\nthey can be um you know uh like the\ndevelopers can intentionally train these\nmodels on very simple on very similar\nexample just to boost the numbers on the\nbenchmarks because\nuh they want to uh ship their product\nand uh they want to show that they have\nincredible performance um and so I truly\nbelieve that as I mentioned that\nBenchmark are very very important and\nthey allow us uh to uh measure and\nunderstand what our models can and\ncannot do uh so we need to do more\nefforts and I'm um uh I'm really uh\nmindblown by the effort of mlc coms\ncongratulation for the new benchmarks\nand it's to play a critical role in the\ncommunity and advancing science around\nsafety um and since I think we're\ntalking about safety here um I don't\nknow probably should I stop here or\nshould I mention something about safety\nbecause today we're talking about safety\nI think lots of people um say that who\ncares why do we care about a model\ngenerating a toxic comment when social\nmedia is flooded with toxic articles and\nall of that and why do we need Benchmark\nlet's figure out how these models reason\nlet's make them more capable in\ngenerating math coding and all of that\nand I think um yeah I mean it makes\nsense so who cares if it can achieve\nyour task successfully but it's actually\na little bit racist I think my concern\nis not there it's it's concerning but\nit's not the uh main uh critical part is\nthat these system can be deployed in\ncritical uh domains and now where\nactually they are being deployed to do\nthings autonomously and that's where the\nwhere it becomes super worrisome and I\nthink Without Really measuring uh the\ncapabilities of these models and the\namount of risk uh they have we can't\nactually deal um uh with more advanced\nsystems and more importantly we cannot\ncontrol them in the future thank you\nthank you yeah we really are learning\nhow to control the behavior of AI\nsystems which I think is just as\nimportant as\ncapability um Yan please uh why are\nbenchmarks important\nyeah I I mean um Dave pson one of my\ncolleagues and during Award winner as\nyou know famously said that For Better\nor Worse Benchmark shape of field and\nthat was not about AI it was in\ngeneral so I think that this tells a lot\nof the importance of a benchmark in any\nfield because people and everyone is is\ngoing to create products in those in\nthat field is going to try to be to show\nwell on that Benchmark there is no\nquestion about that now with AI like was\nsaid earlier it's things are even more\ncomplicated and The Benchmark can be\neven more\nimportant and there are challenging like\norer\ncontamination um and there are\nchallenges about you know safety how you\ndefine uh something is safe or not even\nthat very definition is not is not\ngenerally agreed upon so I think there\nis a lot of it's it's a very important\nproblem we need to make progress on and\nI think there is no question in my in my\nmind and I shouldn't I don't think there\nany question in anyone's mind that benj\nmarar going to be the key which will\nshape the progress of AI the way it\nshapes the progress of other other\nfields thank you one\nC hi Peter um good morning good morning\nyeah I I can hear a little bit of echo I\nthink this should be fine um yeah why\nare benchmarks important of course there\nare one of the very common ways of\ntesting models right now they are of\ncourse the methods like rate teaming and\nso on um but I think benchmarks allow us\nto have a consistent way to compare um\nhow models perform um compare different\nmodels um compare different versions of\nmodel compare them cross time um so and\niterations as well um so these are I\nthink Mets are definitely useful and I\nagree with the um previous panelist to\ntalk about the importance of benchmarks\nfor measuring safety um not withstanding\nI we also recognize that um benchmarks\nso they have many challenges right um\nconcerns about quality\ncomprehensiveness um of various\nbenchmarks are already out there um as\nfar as know there's no perfect Benchmark\nthey\nare even so I think still very useful\nand directionally indicative of um how\nthe models\nsorry um so they can measure specific\ncapabilities they can measure um safety\nand I'm covering um certain areas where\num of improvement for example um so we\ncan identify various specific areas and\nthen then deal with them in terms of I\nbuilding benchmarks to measure these uh\nthese areas so definitely benchmarks\nvery key to um understanding um model\nperformance and behavior\nthanks thank you everyone I I do want to\ndraw attention to one of the I think\nthings that makes us lucky to have this\npanel here today which is we have uh a\nrepresentative of Industry a\nrepresentative of Civil Society the\nnonprofit World a representative of\nAcademia and a representative of the\npolicy World um all you know on a common\nstage to talk about this topic and I\nthink that is part of uh what we're\ntrying to bring together with this work\num so we heard uh everyone talk about\nwhy benchmarks are important um but\nthere's a lot of benchmarks out there so\nmy next question uh again to the panel\nand then we'll ask some more specific\nthings is what are the characteristics\nof a good Benchmark what what is the\nmost important uh you know couple of\nthings that makes a benchmark high\nquality useful whatever\nuh April again we'll start with you yeah\num so to start two things validity and\nreliability and we've covered\nreliability in depth um when I talk\nabout validity um what I mean is is this\nBenchmark useful um is it actually\nmeaningful because one of the things we\ndon't want to do is hyperfixate on a\nmetric that does not actually correlate\nwith the outcomes that we care\nabout in terms of reliability U what I\nmean here is consistency so if you are\nrunning multiple tests um similar\noutputs does that result in uh similar\noutputs um and I think one way to think\nabout quality metrics is also to think\nabout what are the pitfalls for bad\nbenchmarks\num and I think some of these are already\ncovered in um the lightning talks but\nthree come to mind so one I think we've\ntalked about is this problem of\noversaturation so these are scenarios\nwhere we have test sets that are in the\npublic domain that are widely used and\num it's quite easy to develop models\nthat look great against these benchmarks\nbut it's just teaching to the test\ntalked about so um these benchmarks are\nactually hiding a lot of potential\nvulnerabilities and risks um another\npart of it we've talked a lot about\niteration and I think we need to get to\na mindset where this is not one and done\num our understanding of risk is\nevolving um our understanding of the\nmeasurement science is also evolving so\nwe have to continue to iterate and I\nthink the last part that we have not\ntouched on as much is so many of the\nquestions that we're asking are\nquestions of science which obviously is\nimportant um but there's also open\nquestions around policy and governance\nso when we talk about hazards like\nviolence or self harm and then once we\nstart waiting into other territory that\ncan even be more abstract things like\nfairness um these are quite theoretical\niCal and Abstract\nConcepts and what we're trying to do is\nif we think about something like uh\nviolence the taxonomy helps us\nsystematically break down that Concept\nin a way where we can then create\nguidance around allowable and not\nallowable Behavior which then creates\nthe boundaries for what we consider\nacceptable um as we do this for even\nmore abstract Concepts say again\nfairness what we're doing is then we\nhave to build consensus around what we\nconsider is allowable within a\nsystems um so there is a huge part of\nthis where it is the science and making\nsure that we get evaluations right but\nthere's another piece where we have to\nmake sure we're actually tying\ngovernance and policies and how we\nintend to use these systems with\nevaluation thank you April uh newa would\nyou like to expand on that yeah so I\nthink in my view the key criteria IA for\na good Benchmark really depends on the\ntask but there are common criteria that\ndifferent uh benchmarks for different\ntasks should share in my view to have a\nvery good quality data set um I'm going\nto Echo some of the existing work and uh\nwonderful work that has been done in\nthis uh field but also some of my own\nwork uh I guess the first criteria uh is\nsaliency we really need to develop\nbenchmarks um that test relevant\ninformation information that matter\nmatters in real word applications uh to\ngive you an example um I want to\nBenchmark that test for example a\nparticular day in history in World War\nII rather than just a random day in um\nhistory uh and so this is very important\nbecause we don't want just to simulate\nsome scenarios that might be fake and do\nnot really uh represent uh the real word\nand that would be problematic if we are\ntesting our models after um the other\nthing that I think is very very\nimportant and we really need to invest\nmore in time on it is novelty uh we\nreally need to develop benchmarks that\nreveal um Insight uh in models that\nhaven't been captured in previous\nbenchmarks uh and this is important\nbecause um you know our models are\ngetting better and better every day and\nthey're actually performing quite well\nif not uh perfectly on existing\nbenchmarks and this is and this does not\nbring any meaningful um Insight on\nwhether our models are doing better or\nnot and so and also there are new kind\nof skills and risks that are emerging\nwith more capable model that we are not\neven aware of and so by thinking deeply\nabout what kind of capabilities we need\nto come up with um just to give you an\nexample since we're talking about safety\ntoday I think it's it's really good to\num focus on the day-to-day harm and the\nobvious harmful scenarios uh but also we\nhave to think about certain\nthings like copany in language in AI\nmodel copany is uh it means that these\nlanguage models or AI models agreeing\nwith whatever the user says and this is\nbecause these models were trained to be\nhelpful so in order to avoid offending\nuh uh the user they will just agree even\nthough the user might say things that\nare a little wrong not necessarily\nharmful but harmful in a subtle way and\nso this is important the other thing is\nalso misalignment that's also uh very\nimportant to test and we need to think\nmore uh critically uh and creatively\nabout because these models are getting\nmore powerful and so they have now the\nability to uh score very high according\nto your metrics or reward model whatever\nbut at the same time they can diverge\nfrom the developers goal and that's very\nproblematic and we need ways on how to\nmeasure that um uh the other thing that\nI would say before uh besid novelty is\num difficulty and complexity of the task\nwe really need to think about about how\nwe can develop harder uh and difficult\nbenchmarks um we have to really avoid\nthe problem of uh Benchmark saturation\nuh you know right now we can develop a\nbenchmark and within a few days even\nweeks um these models can score very\nhighly there and so uh we know that you\nknow solving as I mentioned earlier\nsolving the data set doesn't mean that\nyou know the model really develop the\nunderlying skill to solve a particular\ntask we have have to all the time keep\niterating on the data set and this\nbrings me to the other one I think uh\nDynamic data set we really need to move\naway from a static data set and this has\nbeen the case in the uh machine Learning\nCommunity for years now and uh in my\nopinion similar to how we actually write\ncode and we push the code and create\npool request and we refactor our code it\nshould be the same practice that we do\nfor these data sets uh we should be able\nto dynamically uh improve or upgrade and\nalso it's okay to deprecate certain data\nset we say okay this data set is super\neasy the model is performing quite well\non it it doesn't provide any new insight\nto us so let's move on let's upgrade it\nor let's decide to completely get rid of\nit um uh what other thing obviously uh\nwe have to uh um make it also multi-turn\nuh single turn is great but as a as a\nstart but it doesn't necessarily capture\nuh the multiple interaction that happen\nwith the user so we might be missing on\ncertain important skills um and I think\nhere probably if I want to be more\nspecific towards having a more um um you\nknow Universal AI model is that we have\nmultiple millions of user using it\naround the word and so I think uh The\nBenchmark needs to take into account\ncertain criteria like uh uh\npluralistic\num uh uh like participatory like the\ndata set should actually test different\nValu for different uh background\ndifferent culture and different\nlanguage thank you and I I think I\nreally like your point highlighting the\nimportance of data there's a whole\ndiscussion a whole different event we\ncould do uh plunging deep into uh to AI\ndata Yan um what would you add about uh\nhow you build a good Benchmark yeah it's\nnot much about adding it's maybe\nemphasizing some of the things which you\nalready said um one it's obviously to\navoid contamination that's very clear\nthat models are going to are trained on\nall the possible data they can find the\ndata is a bottleneck so of course they\nare going to be trained all the pageb\nthemselves if they are available um\nsometimes even without knowing um so\nthat's number one the number two is a\nbenchmark you know\nsaturation so I think that you always\nwant to you know like as the models\ngrows into uh sophistication and then\nyou know they're going to do well on\nmost Ben marks once you are going to be\nat 90% and so forth you are all\nsaturated um you are going to talk only\nabout you know 1% to% um so I think\nthat's for instance at on chatbot Arena\nwe have we identified the question a set\nof question which you called Arena hard\nand there are harder question which kind\nof try to increase a Delta\nbetween uh you know the models which are\nsimilar in power on the general\nbenchmarks\num so that's you know the second point\nthe third point I think is very\nimportant to to consider that um all a\nlot of this a lot of this is not only\nabout math questions or what you where\nis where is where is clearly what is you\nknow the right answer or the wrong\nanswer right but it's about how what\npeople feel it's a good answer or not\nright so actually having the people in\nthe loop doing the evaluation is\nextremely important and people say and\nactually you know say okay but this is\nis you know have a lot of things people\ncomplaining and oh the style of the\nmodel you know because this this kind of\nthe stylistic is better users are\npreferring it and we did all these\nstudies and you know and we can actually\num account for the stylistic differences\nwe can factor out the bias uh and\nbasically by by the way a parenthesis\nhere once you know the what if a bias if\nyou have enough data you can factor out\nthe bias so actually it's not the\nproblem is not the biases you know the\nproblem is about the biases you do not\nknow because that's how you cannot\nfactor out but anyway but you know so we\ncan do that and actually if you look at\nthe the latest incarnation of the\ncharbot Arina you can select to factor\nout these biases the why Vibe and things\nlike that but the truth is that you know\nthe the People Are People right so they\nlike some kind of sty style better than\nthe other one so it's good to know that\nas well right and\nwhat is safe or not it depends also on\nthe cultural background and things like\nthat right is something it's offensive\nin us maybe not offensive in whatever I\nknow I'm not going to say in another\ncountry but you can imagine that right\nso I think this kind of cultural\nDimension is also extremely important so\nI think that you need when we are\ntalking here not about one one Benchmark\nit's a kind of a battery of benchmarks\nfor different tasks for you know for\ndifferent culture and so forth and the\nlast point I want to make is\nthat I also thought that in some sense\nyou know um testing you know\nbenchmarking these models is not unlike\nuh testing people right if you look at\npeople and how we we test people it's\nvery similar right we try to avoid\ncontamination or contamination means for\npeople oh you know the problems in the\nexam before the exam you take the exam\nright that's contamination right so you\nalways come you come to come with new\nproblems in every exam every exam it's\nin you know once and done and um the\nsame thing you are going to have exams\nfor different fields for different\nsubjects and um I think it's it's a\nproblem which you are dealing you know\nI'm talking here as a faculty right it's\nlike is a final you know the next the\nthis is the final week of classes the\nnext week is are the exams right so it's\nit's very so I think that getting some\ninspiration for how we as a humans\nevaluate humans um is not a you know\nit's it's it's something to look at um\nso I'll stop here for now thank\nyou want say I particularly from a\npolicy perspective what are the\ncharacteristics of a good\ntouchine that's a very good question\nthat we're always asking ourselves\nactually um I wish we have\nuh uh guide or methodology this is how\nyou build a benchmark right or this is\nhow you assess if a benchmark is good\nand talking about Benchmark assessing\nmodels we also need to assess benchmarks\num from a policy perspective um\ndefinitely I think U we always comes\nfrom what are the safety concerns that\nwe want to measure first right and then\nare we actually actually measuring these\num safety concerns whether it's from a\ntaxonomy of um safety issues that we've\nidentified or um certain use cases in\nuse cases and context so uh are the bpar\nactually measuring um these issues are\nthey representing um the range of users\nthat going to be engaging with the model\ndealing with these issues um is a data\nset that's been used to generate The\nBenchmark um um put together in in a way\nthat doesn't introduce any buyers or\nother concerns or other problems um one\nof the most common questions I get asked\nis is how many questions would there be\nthat said right talking about Benchmark\nshould we be asking a thousand questions\n100,000 questions and I think it's\nhighly also dependent on how much we\nmeasure um someone told me a th for one\ntopic maybe the the big rule that we can\nstick to but I I don't know I don't\nthink we have landed on that clearly yet\num so there are all of these issues that\nthe previous panelists have highlighted\nas well right making sure that we think\nabout um um over\nsaturation there's no\ncontamination um so we also had a lot\ndiscusss around public and private\nbenchmarks should benchmarks be public\ncompletely there there usefulness there\ncertain use to having public benchmarks\nthen you know more people can use them\nthey can um um um actually run different\nmodels and then they can improve on them\nas well and understand um where where\nthe current state of the artst but at\nthe same time we worry about you know as\nprevious panelists have already said mot\num being trained on these benchmarks so\nquite a lot of um discussions on um\nbenchmarks and I think maybe one of the\nother areas that we s talk about then is\nhow do we measure what what the metrics\num that we want to put in place um in\nmeasuring in in identifying performance\nso it's\nalso um um\nwhat are some of the scoring mechanisms\num open and open-ended and close-ended\num questions and answers um as well as\num as someone said earlier having\nputting in place the human um to\nactually look at the answers and say\nthis is this is right or wrong as as\ninitial set of understanding um where we\nstand um agree also with panelist on um\ndifferent values so how then do we\nactually look at the result um from a\nsafety Benchmark especially right um\nValu are important and then um being\nable then to understand where is how how\nwhat will be acceptable in um different\ncultures um when it comes to um the\nresults so these are various areas um\nvarious issues relating to benchmarks\nlike I said at the beginning I if\nsomeone writes a good this is a\nmethodology for building benchmarks I\nthink that be super useful for all of us\nthanks U I think uh it's interesting to\nsee the methodology evolved I think the\nhidden test practice test approach uh we\ntook in AI Luminate hopefully can can\nhelp with some of the things you're\ndescribing the balance of transparency\nand uh uh resistance to cheating uh I\nask a couple of specific questions and Y\nI'd like to to start with you um capot\nArena has become a very popular\nBenchmark I think in part because people\ncan understand it uh you ask uh you give\na prompt to two different systems and\nyou pick the one you like better and uh\nthat system gets a gets a point so to\nspeak and and that uh results in a\nleaderboard um could you talk a little\nbit about the importance of uh\ntransparency and understandability\nespecially for non-experts people who\naren't you know all of us thinking all\nthe time about uh safety and and\nbenchmarking yeah so um yeah sorry there\nis a little a little bit of echo uh\nwhich is\nuh um uh so yeah we shat abouta like you\nsaid it's um it's a in the wild\nBenchmark um you go to the site you ask\na question and we provide uh answer from\ntwo anonymized models um is multi- turn\nyou can go multi-turn and then you can\npick which uh answer you believe is\nbetter uh or maybe whether both both are\ngood or both or both are\nbad and uh then we inspire originally\nyou know like of course I'm not going to\ngo into details um but we inspire you\nknow then we to we comput a rating which\nis inspired from you know chess right\nwhere um chess players are rated using\nElo Elo rating and basically in some\nsense if you beat a stronger player you\nare going to get a bigger bump than if\nyou beat a weaker player player right\nand this is uh you know kind of a very\nnice model because the models you know\nthere are um you you you you cannot play\na model for every for the same question\nagainst every other model um and this\nkind of um uh rating is it's also pretty\nDynamic um it can change in time and\nthings like that so um I I found that\nit's it was a good model uh and again we\ndon't take as much credit we just kind\nof copyed what we saw look around and we\nsaw kind of is working in other\nfields um and then what what happens\nactually right now is that uh you know\nyou people ask more and more questions\nso see becomes more and more um\ninteresting and um like for instance\nfirst we have only one score right\noverall score\nnow we have scores by\ncategories math and um reasoning English\nby by language and things like that um\nthen like I mentioned earlier people\nsaid you know very you know uh it was\nvery good question hey but you know look\nit's like this is uh people prefer this\nuh model because provide the answer is\nbetter formatted right than the other\none right um so we with or or the\nlanguage is kind of you know some many\ntimes the people prefer actually the\nmore veros answers okay uh many times Al\nthey prefer the first answer right so\nyou looking a lot of a lot of time in\nthis kind of bias coming from the people\num and try to factor it out right so now\nyou can look and say okay if you want to\num factor out the kind of formatting or\nsomething like that you can see uh you\nknow how the model that model uh uh the\nranking changes and it does change right\nso so and then you have also obviously\nmultimodality right now it's a lot of\ntext to image or uh you know um and text\nvideo and like that and then you have a\nlittle bit more even more kind of task\noriented right we have right now you\nknow for instance for uh how for\nco-pilot it's a popular application how\ndo you compare those right how which one\nis better right we have have now a\nversion you which basically you can\nbasically it's done with some colleagues\nfrom CMU actually it's um they lead this\nwork you know which you you know you you\nuse is is integrated with visual code\nand um you get now instead of one\nsuggestion for a copilot you have two\nsuggestions and you keep keep you pick\none of those\num so what we and and by the way the\nparenthesis here it's a it's a very\nstrong uh position bias people prefer\nthe first answer um uh so what we are\nseeing here is\nthat um it's it's very complex the space\nis is hugely complex uh so I think that\nuh going back on one see on one she said\num I think that it's it's almost it's\nextremely important to try to figure F\nout what makes a good benchmarks what\nare the rules for a good Benchmark\nbecause you are going to have a lot of\nbenchmarks right because they are so\nGeneral right um these models so um yeah\nso we learn a lot in the process and um\nand there are still a lot of questions\nwhich needs to be answered like for\ninstance freshness it's another question\nright like we are because this in the\nwild we are still releasing some 20% of\nthe answers and the votes right and we\nwe don't release more because we want to\ntry to avoid uh um overfitting right um\nbut you know is that this will this is\nvaluable only if it's almost you always\nkind of you have a\ncertain number of question of prompts\nare all are are fresh right right so is\nthat the case and what is that kind of\nyou know the number of fresh questions\neven what a fresh question it's it's\ninteresting it's interesting question\nright no pun intending right how you\nknow that oh this question is is really\na different questions right you know\nit's like so it's it's it's it's a it's\nhuge space it's in some ways the essence\nof good benchmarking is we're trying to\ntake these incredibly complex problems\nand distill them down into\ninformation yeah so and\nand yeah go ahead sorry I just want to\nuh stop you there y because I I want to\nmake sure we get through a couple more\nmore answers in our our uh very limited\nuh remaining time so I'm going to going\nto do a you know the lightning answer\nround um so newa one of the things I\nwanted to uh to point to um with with\nyour work at uh ai2 um is the omo models\nand we used an older version of an omo\nmodel as essentially a control in our\ninitial testing we wanted something that\nwas a legitimate model but was older and\nnot safety to\nto show frankly that it wouldn't do so\nwell on a benchmark like this\num but going forward we'd love to work\nwith organizations like yours to develop\nvery open sort of gold standard\nreference models that everyone's\nunderstands and use those to help set\nour curve could you talk a little bit\nabout um omo 2 and uh your your plans in\nthis direction yeah yeah very good\nquestion um I think uh the main motto uh\nAlo 2 and ai2 is openness and\ntransparency so uh we're not just being\nopen we're being more than open so we're\nreleasing models we're releasing data\nwe're releasing every single detail of\nthe pipeline development of these AI\nmodels from the simple hyperparameter\nsearch to how we collected the data what\nkind of Training Method we use and I\nthink that um by creating this\ntransparent and publicly acceptable\nresources to um uh we can actually help\nevaluating models and also comparing uh\nthe performance of these models and this\nis uh particularly important because um\nwe can that way understand why certain\nmodels performed this well on this\nBenchmark or poorly on this Benchmark\nand I know that there are lots of um\nmodels already in the community that are\nopen weights but we're still lacking the\ndata that they were trained on and I\nthink that's uh we we know that data\nplays a critical role in the performance\nof the model like yes having access to\nthe model weights uh it's not enough\nit's great but it's not enough to\nunderstand what has led to a certain\nBehavior so we need to do that and I\nthink that um you know openness uh will\nprovide transparency and scrutiny and\nmost importantly the uh understanding of\nthe inter working of these uh models and\num it's it's really challenging and this\nis why we're being open because because\nI think that you know the achieving\ncertain goals in advancing AI models uh\nis not the job of a one single Institute\nor a lab but it's more like a collective\neffort towards um figuring out the main\nissues um and so we're trying to empower\nthe entire Community to perform and\nconduct research and advanced uh science\naround these models so we're hoping with\nAI too uh with Alo to um with also two\nthree uh models uh people will get to\nunderstand better how to build these\nmodels and most importantly how to\nmonitor them and how uh to make them\nsafe and to advance them um and so\nyeah we really look forward to uh to\nworking more closely with you to try and\ncreate like the uh the gold kilogram\nunderglass you know for AI safety um and\nthat openness is really helpful there\num ony uh I wanted to ask you how you\nsee um increased Global cooperation we\ntalk about trying to make a global\nstandard as being really important in\nthe AI benchmarking\nspace yeah definitely I think we spoke a\nlittle bit about different values and\ndifferent cultures having um different\nexpectations for example\num so having a common set of benchmarks\nfor us to look at together I think\nthat's uh that a good start um I think\nthe um taxonomy that's been defin uh is\nuseful for at least for the first\nconversation does this make sense to\neveryone around the world right um I\nthink I spoke um I think one of our\nearlier events together P that talked\nabout finding that Universal set of\nconcerns that um everyone has I think is\nuseful um so Global Corporation in\nhaving this discussions that Ely\nimportant and then identifying where\nit's specific um to cultures and it's\nnot just in terms of the questions that\nask but also in understanding the\nresponses that been generated Through\nThe Benchmark some may be acceptable to\nsome cultures and some may not be right\nso having that understanding is\nimportant so the global coroporation to\nto get to that level of um of detailed\nunderstanding will be useful and the\nother area in which to cooperate is\nreally then in making sure that these\nbenchmarks are are available in\ndifferent languages um that can then\nagain um not just test for performance\nin different languages but also um test\nfor whether you know some safeguards\nbreak down in different languages when\nit comes to um um certain models right\nso um so we are quite happy to be um\nworking together in this area I mean\nfrom Singapore there are very\nfoundations helping um with you know\ndeveloping um evaluators in different\nlanguages actually to evaluate the\noutput it's not just about translating\nquestion but also looking at output and\nbeing able to evaluate output on uh on\nwhether these are um these are safe or\nunsafe score them uh across a different\nset set of um tricks um so many things\nthat we can do together actually\nglobally as a community um to help um\nyeah improve safety and I think starting\nwith actually looking at the benchmarks\ntogether addressing if questions of your\nevaluators these are these are good ways\nto start\nthanks so last last question goes to you\nApril um in particular I think safety is\noften presented as as either being good\nfor society or you know a burden on\nindustry do you think that is true how\ndo you see you know this work as being\nuh potentially\nbeneficial yeah I think there's been a\nfalse narrative when it hits safety\nagainst\nInnovation um when in fact I think one\ninces the\nother so\nwhat we're focused on is we're focused\non adoption so we want to build\ntechnologies that are\nuseful and that are\ntrustworthy in order to to build that\ntrust we need to be able to be\ntransparent about what our models are\ncapable of doing um the potential gaps\nthat still exist and the potential\nrisks so th this happens\nuh across a number of different areas so\nas a model provider what we need to do\nis we need to focus on some foundational\nand Broad understandings around safety\nand we need to make that transparent and\nthen for those who are model adopter so\nusers developers businesses and\norganizations they need to take this\ninformation that's being provided and\ndevelop contact specific and use case\nspecific guard rails and mitigation\nstrategies because until we get to a\nplace where people feel confident that\nthese models are safe that these models\nare trustworthy it's not going to be a\ngood product experience for\nusers so\nultimately safety and adoption they go\nhand inand and those together\nAdvanced Innovation for real use cases\nwhich is how we're going to live the\npromise of this\ntechnology thank you very much April uh\none last uh quick question from uh the\naudience uh about uh the training data\nfor AI so you're releasing the training\ndata um so with that we're going to wrap\nup our panel and thank you very much\neveryone one for the the context uh you\nprovided um really great to see these\ndifferent perspectives industry Civil\nSociety Academia policy uh coming\ntogether on this very important question\nthat is what we are are trying to do\nwith this Benchmark in many ways and it\nmay not be obvious in AI the test is the\ndesign um the the test for safety if we\nall agree on a standard uh it gives a\nconsistent definition of safety for\npeople to build towards for people to\nbuy with confidence um there's a lot of\nvalue at that um it refines questions\nlike what exactly are we testing is it\nthe model or the model plus guard rails\nI know some of the systems we tested uh\nnow they're releasing guard rails Mr\ndraws out with uh with theirs um meta\nhas uh has very clear guardrails in in a\nDocker stack so we're really beginning\nto answer uh these questions about how\nyou build composite systems um and what\nyou should expect from them this is\nthese are the kind of questions that we\ncan answer with a standard Benchmark and\na standard methodology so once again\njust wanted to give a great uh thank you\nto everyone who's helped us our\npanelists today our speakers um everyone\nin the ml Commons Community who's come\ntogether uh to help us uh build and\nlaunch Benchmark and start this this\nprocess of defining a production\nperspective uh on AI safety thank you\nall thank you everyone for coming today\n[Applause]",
  "duration_seconds": 7214,
  "is_auto_generated": true,
  "transcript_source": "youtube_transcript_api",
  "transcript_status": "available",
  "asr_quality_weight": 0.6,
  "transcript_segments": [
    {
      "text": "good morning everyone my name is Rebecca",
      "start": 3,
      "duration": 4
    },
    {
      "text": "Weiss and I'm the executive director of",
      "start": 5.279,
      "duration": 4.4
    },
    {
      "text": "ml coms sorry that I couldn't join you",
      "start": 7,
      "duration": 4.12
    },
    {
      "text": "all in person today but it is my",
      "start": 9.679,
      "duration": 2.92
    },
    {
      "text": "pleasure to welcome you to the launch of",
      "start": 11.12,
      "duration": 3.479
    },
    {
      "text": "our AI Luminate",
      "start": 12.599,
      "duration": 4.121
    },
    {
      "text": "FR for those of you who may not be",
      "start": 14.599,
      "duration": 4.68
    },
    {
      "text": "familiar ml commin is an open AI",
      "start": 16.72,
      "duration": 4.24
    },
    {
      "text": "engineering Consortium that is built on",
      "start": 19.279,
      "duration": 3.92
    },
    {
      "text": "the belief that trustworthy safe and",
      "start": 20.96,
      "duration": 4.559
    },
    {
      "text": "efficient AI depends on better systems",
      "start": 23.199,
      "duration": 3.681
    },
    {
      "text": "for measur",
      "start": 25.519,
      "duration": 4.68
    },
    {
      "text": "accountability you can't trust let alone",
      "start": 26.88,
      "duration": 6
    },
    {
      "text": "can't test and we believe that when we",
      "start": 30.199,
      "duration": 5.121
    },
    {
      "text": "start measuring things they tend to",
      "start": 32.88,
      "duration": 4.72
    },
    {
      "text": "improve we are also grounded in the",
      "start": 35.32,
      "duration": 4.28
    },
    {
      "text": "philosophy that to improve AI just like",
      "start": 37.6,
      "duration": 4.639
    },
    {
      "text": "any other type of complex technology we",
      "start": 39.6,
      "duration": 5.16
    },
    {
      "text": "must work together no single",
      "start": 42.239,
      "duration": 4.401
    },
    {
      "text": "organization or entity has all of the",
      "start": 44.76,
      "duration": 4.36
    },
    {
      "text": "answers instead this work can only be",
      "start": 46.64,
      "duration": 4.439
    },
    {
      "text": "done by leveraging the unique expertise",
      "start": 49.12,
      "duration": 4.88
    },
    {
      "text": "and perspective of Academia industry and",
      "start": 51.079,
      "duration": 4.441
    },
    {
      "text": "Civil",
      "start": 54,
      "duration": 3.879
    },
    {
      "text": "Society the first step of reaching a",
      "start": 55.52,
      "duration": 4.839
    },
    {
      "text": "goal is agreeing on what that goal is",
      "start": 57.879,
      "duration": 4.961
    },
    {
      "text": "that's why since 2018 we have brought",
      "start": 60.359,
      "duration": 4.561
    },
    {
      "text": "together academic and Industry resources",
      "start": 62.84,
      "duration": 5.319
    },
    {
      "text": "to create trusted empirically sound AI",
      "start": 64.92,
      "duration": 4.28
    },
    {
      "text": "performance",
      "start": 68.159,
      "duration": 3.361
    },
    {
      "text": "maners and now with roots and",
      "start": 69.2,
      "duration": 4.72
    },
    {
      "text": "well-respected research institutions an",
      "start": 71.52,
      "duration": 5.239
    },
    {
      "text": "open and transparent process and buy and",
      "start": 73.92,
      "duration": 4.879
    },
    {
      "text": "across the industry we are tackling the",
      "start": 76.759,
      "duration": 3.961
    },
    {
      "text": "Urgent issue of AI",
      "start": 78.799,
      "duration": 4.28
    },
    {
      "text": "safy so today you're going to hear much",
      "start": 80.72,
      "duration": 5.32
    },
    {
      "text": "more about AI Lumin this is ml Common's",
      "start": 83.079,
      "duration": 4.801
    },
    {
      "text": "new Benchmark for measuring the risk",
      "start": 86.04,
      "duration": 4.759
    },
    {
      "text": "reliability of large langage",
      "start": 87.88,
      "duration": 5.04
    },
    {
      "text": "this is a major Milestone not just for",
      "start": 90.799,
      "duration": 4.241
    },
    {
      "text": "ML Commons but for our Global community",
      "start": 92.92,
      "duration": 4.32
    },
    {
      "text": "of academics industry leaders and",
      "start": 95.04,
      "duration": 4.759
    },
    {
      "text": "Advocates that are dedicated to",
      "start": 97.24,
      "duration": 5.4
    },
    {
      "text": "saf to our partners our supporting",
      "start": 99.799,
      "duration": 4.441
    },
    {
      "text": "companies and of course all of the",
      "start": 102.64,
      "duration": 3.159
    },
    {
      "text": "members of our risk and reliability",
      "start": 104.24,
      "duration": 3.839
    },
    {
      "text": "working group all of whom made this Happ",
      "start": 105.799,
      "duration": 5.081
    },
    {
      "text": "thank you so much I'd also like to thank",
      "start": 108.079,
      "duration": 4.36
    },
    {
      "text": "the incredibly talented members of our",
      "start": 110.88,
      "duration": 4.12
    },
    {
      "text": "ml Commons team this achievement would",
      "start": 112.439,
      "duration": 5.441
    },
    {
      "text": "not have been possible without so we are",
      "start": 115,
      "duration": 5.439
    },
    {
      "text": "thrilled to share a with you today and",
      "start": 117.88,
      "duration": 4.199
    },
    {
      "text": "we look forward to continuing to partner",
      "start": 120.439,
      "duration": 3.561
    },
    {
      "text": "with you as we develop and Advance a",
      "start": 122.079,
      "duration": 3.68
    },
    {
      "text": "global",
      "start": 124,
      "duration": 4.16
    },
    {
      "text": "basine with that I'm pleased to welcome",
      "start": 125.759,
      "duration": 6.321
    },
    {
      "text": "ml comm's founder and president",
      "start": 128.16,
      "duration": 3.92
    },
    {
      "text": "Peter good morning everyone thank you",
      "start": 135.48,
      "duration": 5.68
    },
    {
      "text": "all very much for coming a few quick",
      "start": 138.04,
      "duration": 5.4
    },
    {
      "text": "Logistics uh please turn off your cell",
      "start": 141.16,
      "duration": 4
    },
    {
      "text": "phone or mute your cell phone if you",
      "start": 143.44,
      "duration": 3.439
    },
    {
      "text": "haven't already and if you need it the",
      "start": 145.16,
      "duration": 4.439
    },
    {
      "text": "restrooms are that way I want to start",
      "start": 146.879,
      "duration": 5.521
    },
    {
      "text": "this morning with a quick uh crowd check",
      "start": 149.599,
      "duration": 4.321
    },
    {
      "text": "so please raise your hand if you think",
      "start": 152.4,
      "duration": 3.16
    },
    {
      "text": "AI safety is",
      "start": 153.92,
      "duration": 4.48
    },
    {
      "text": "important all right great please raise",
      "start": 155.56,
      "duration": 4.399
    },
    {
      "text": "your hand if you think we have ai",
      "start": 158.4,
      "duration": 4
    },
    {
      "text": "evaluations figured",
      "start": 159.959,
      "duration": 6.481
    },
    {
      "text": "out all right make the rest of this talk",
      "start": 162.4,
      "duration": 7.68
    },
    {
      "text": "easier so I'm presenting today uh AI",
      "start": 166.44,
      "duration": 5.079
    },
    {
      "text": "Luminate version",
      "start": 170.08,
      "duration": 4.079
    },
    {
      "text": "1.0 uh which we view as a first step",
      "start": 171.519,
      "duration": 4.841
    },
    {
      "text": "towards a global standard for AI product",
      "start": 174.159,
      "duration": 5.681
    },
    {
      "text": "safety this is work by as Rebecca was",
      "start": 176.36,
      "duration": 6
    },
    {
      "text": "was saying a lot of talented people uh",
      "start": 179.84,
      "duration": 5.039
    },
    {
      "text": "in particular the ml Commons AI risk and",
      "start": 182.36,
      "duration": 5.439
    },
    {
      "text": "reliability working group and the AI",
      "start": 184.879,
      "duration": 5.521
    },
    {
      "text": "verifi",
      "start": 187.799,
      "duration": 2.601
    },
    {
      "text": "Foundation I want to start by talking",
      "start": 190.879,
      "duration": 6.041
    },
    {
      "text": "about AI progress and the challenges we",
      "start": 193.36,
      "duration": 4.599
    },
    {
      "text": "face",
      "start": 196.92,
      "duration": 5.679
    },
    {
      "text": "today so for a long time decades AI was",
      "start": 197.959,
      "duration": 6.801
    },
    {
      "text": "a bunch of very cool ideas that never",
      "start": 202.599,
      "duration": 5.081
    },
    {
      "text": "quite worked but now we've entered a new",
      "start": 204.76,
      "duration": 5.16
    },
    {
      "text": "era I'm going to describe as the era of",
      "start": 207.68,
      "duration": 5.08
    },
    {
      "text": "amazing research and scary headlines and",
      "start": 209.92,
      "duration": 6.08
    },
    {
      "text": "in order to get there we had to break",
      "start": 212.76,
      "duration": 5.52
    },
    {
      "text": "through a barrier capability barrier and",
      "start": 216,
      "duration": 4.56
    },
    {
      "text": "we did that with Innovations like deep",
      "start": 218.28,
      "duration": 4.879
    },
    {
      "text": "neural networks um like",
      "start": 220.56,
      "duration": 6.959
    },
    {
      "text": "Transformers uh and benchmarks uh like",
      "start": 223.159,
      "duration": 5.36
    },
    {
      "text": "uh",
      "start": 227.519,
      "duration": 6.041
    },
    {
      "text": "imet but today we want to reach a",
      "start": 228.519,
      "duration": 8.321
    },
    {
      "text": "third and that is the era of products",
      "start": 233.56,
      "duration": 6.879
    },
    {
      "text": "and services that deliver real value to",
      "start": 236.84,
      "duration": 6.64
    },
    {
      "text": "users to businesses and to Society at",
      "start": 240.439,
      "duration": 5.72
    },
    {
      "text": "large and in order to get",
      "start": 243.48,
      "duration": 6.44
    },
    {
      "text": "there we need to pass through another",
      "start": 246.159,
      "duration": 7.64
    },
    {
      "text": "barrier a risk and reliability barrier",
      "start": 249.92,
      "duration": 5.959
    },
    {
      "text": "imagine the best AI demo you've ever",
      "start": 253.799,
      "duration": 4.601
    },
    {
      "text": "seen now imagine an AI system that could",
      "start": 255.879,
      "duration": 6.2
    },
    {
      "text": "do that every single time in real world",
      "start": 258.4,
      "duration": 6.4
    },
    {
      "text": "conditions with very low risk that is",
      "start": 262.079,
      "duration": 5.601
    },
    {
      "text": "where we are trying to",
      "start": 264.8,
      "duration": 6.52
    },
    {
      "text": "go we've seen this before or in other",
      "start": 267.68,
      "duration": 6.84
    },
    {
      "text": "Industries so if you look at um Aviation",
      "start": 271.32,
      "duration": 5.08
    },
    {
      "text": "for instance you can look all the way",
      "start": 274.52,
      "duration": 4.2
    },
    {
      "text": "back to the sketchbooks of Leonardo D",
      "start": 276.4,
      "duration": 4.639
    },
    {
      "text": "great ideas that never quite worked and",
      "start": 278.72,
      "duration": 5.08
    },
    {
      "text": "then you see the breakthroughs that make",
      "start": 281.039,
      "duration": 5.361
    },
    {
      "text": "them prac possible um like the Wright",
      "start": 283.8,
      "duration": 5.16
    },
    {
      "text": "brothers at Kittyhawk but there was a",
      "start": 286.4,
      "duration": 4.799
    },
    {
      "text": "tremendous amount of work from that",
      "start": 288.96,
      "duration": 4.72
    },
    {
      "text": "first flight to",
      "start": 291.199,
      "duration": 5.84
    },
    {
      "text": "the almost unbelievably safe uh",
      "start": 293.68,
      "duration": 6.48
    },
    {
      "text": "commercial Aviation and on today",
      "start": 297.039,
      "duration": 4.841
    },
    {
      "text": "many of us in this room wouldn't be here",
      "start": 300.16,
      "duration": 4.96
    },
    {
      "text": "if not for all the work and the",
      "start": 301.88,
      "duration": 5.84
    },
    {
      "text": "measurement uh that enabled that",
      "start": 305.12,
      "duration": 7.12
    },
    {
      "text": "progress to a highly reliable lowrisk",
      "start": 307.72,
      "duration": 8.479
    },
    {
      "text": "service to get here for AI we need",
      "start": 312.24,
      "duration": 7.2
    },
    {
      "text": "standard AI safety",
      "start": 316.199,
      "duration": 6.361
    },
    {
      "text": "benchmarks we've seen benchmarks Drive",
      "start": 319.44,
      "duration": 6.039
    },
    {
      "text": "progress across Industries whether it's",
      "start": 322.56,
      "duration": 7.6
    },
    {
      "text": "new cars uh safety of automobiles uh",
      "start": 325.479,
      "duration": 6.72
    },
    {
      "text": "Environmental Quality of buildings",
      "start": 330.16,
      "duration": 3.159
    },
    {
      "text": "Hospital",
      "start": 332.199,
      "duration": 3.881
    },
    {
      "text": "quality we've also seen this firsthand",
      "start": 333.319,
      "duration": 6.88
    },
    {
      "text": "at ml coms so ml commom got its Genesis",
      "start": 336.08,
      "duration": 6.36
    },
    {
      "text": "from the ml perf",
      "start": 340.199,
      "duration": 4.881
    },
    {
      "text": "Benchmark so ml perf is a measure of",
      "start": 342.44,
      "duration": 6.039
    },
    {
      "text": "Hardware speed and software sack speed",
      "start": 345.08,
      "duration": 4.92
    },
    {
      "text": "and what we've seen since the",
      "start": 348.479,
      "duration": 4.521
    },
    {
      "text": "introduction of that Benchmark is a 50x",
      "start": 350,
      "duration": 5.96
    },
    {
      "text": "speed up in five years so this this",
      "start": 353,
      "duration": 4.88
    },
    {
      "text": "graph shows different parts of that",
      "start": 355.96,
      "duration": 4.4
    },
    {
      "text": "Benchmark getting faster over time on a",
      "start": 357.88,
      "duration": 5.319
    },
    {
      "text": "logarithmic scale the dark line at the",
      "start": 360.36,
      "duration": 5.76
    },
    {
      "text": "bottom is Mo's law so this is an",
      "start": 363.199,
      "duration": 5.681
    },
    {
      "text": "incredibly fast",
      "start": 366.12,
      "duration": 6
    },
    {
      "text": "ramp and the question is can we do this",
      "start": 368.88,
      "duration": 5.879
    },
    {
      "text": "for sa this is what we're trying to",
      "start": 372.12,
      "duration": 5.919
    },
    {
      "text": "achieve with AI",
      "start": 374.759,
      "duration": 3.28
    },
    {
      "text": "Luminate AI Luminate is aimed at what we",
      "start": 379.08,
      "duration": 5.44
    },
    {
      "text": "describe as AI product",
      "start": 382.4,
      "duration": 4.6
    },
    {
      "text": "safety so it's worth talking about what",
      "start": 384.52,
      "duration": 4.64
    },
    {
      "text": "that means because AI safety means a lot",
      "start": 387,
      "duration": 4.319
    },
    {
      "text": "of things a lot of different people",
      "start": 389.16,
      "duration": 5.92
    },
    {
      "text": "product safety is is hazards from users",
      "start": 391.319,
      "duration": 6.32
    },
    {
      "text": "of AI systems or hazards to users of AI",
      "start": 395.08,
      "duration": 5.72
    },
    {
      "text": "syst near-term",
      "start": 397.639,
      "duration": 6.721
    },
    {
      "text": "practical business value oriented that's",
      "start": 400.8,
      "duration": 5.92
    },
    {
      "text": "product there are other important",
      "start": 404.36,
      "duration": 4.08
    },
    {
      "text": "aspects of AI safety that other folks",
      "start": 406.72,
      "duration": 4.08
    },
    {
      "text": "are looking at this includes Frontier",
      "start": 408.44,
      "duration": 3.56
    },
    {
      "text": "safety so this is what happens if the",
      "start": 410.8,
      "duration": 4.839
    },
    {
      "text": "models get too smart economy um Advanced",
      "start": 412,
      "duration": 6.84
    },
    {
      "text": "uh Advanced cbrn hazards things like",
      "start": 415.639,
      "duration": 4.801
    },
    {
      "text": "that",
      "start": 418.84,
      "duration": 2.88
    },
    {
      "text": "and there's also",
      "start": 420.44,
      "duration": 4.199
    },
    {
      "text": "social there are real economic and",
      "start": 421.72,
      "duration": 4.28
    },
    {
      "text": "environmental risks around the",
      "start": 424.639,
      "duration": 3.161
    },
    {
      "text": "development and deployment of AI and",
      "start": 426,
      "duration": 3.12
    },
    {
      "text": "these are conversations that we should",
      "start": 427.8,
      "duration": 3.799
    },
    {
      "text": "be having but what we are focused on is",
      "start": 429.12,
      "duration": 4.44
    },
    {
      "text": "that",
      "start": 431.599,
      "duration": 4.72
    },
    {
      "text": "producty and in particular in this first",
      "start": 433.56,
      "duration": 5.4
    },
    {
      "text": "version this first Benchmark of what we",
      "start": 436.319,
      "duration": 5.44
    },
    {
      "text": "plan to be a growing family of",
      "start": 438.96,
      "duration": 5.84
    },
    {
      "text": "benchmarks we are focused on general",
      "start": 441.759,
      "duration": 5.241
    },
    {
      "text": "purpose chat",
      "start": 444.8,
      "duration": 4.48
    },
    {
      "text": "systems so these are systems that you",
      "start": 447,
      "duration": 4.28
    },
    {
      "text": "inter interact with just this is chat",
      "start": 449.28,
      "duration": 3.68
    },
    {
      "text": "GPT you interact with it by asking it",
      "start": 451.28,
      "duration": 3.319
    },
    {
      "text": "questions asking it to help you do",
      "start": 452.96,
      "duration": 3.6
    },
    {
      "text": "things most of which are good things",
      "start": 454.599,
      "duration": 4.16
    },
    {
      "text": "some of which are not uh so you ask it",
      "start": 456.56,
      "duration": 3.96
    },
    {
      "text": "how to to kill your boss and not get",
      "start": 458.759,
      "duration": 4.321
    },
    {
      "text": "caught um the system considers that and",
      "start": 460.52,
      "duration": 3.88
    },
    {
      "text": "it's going to give you one of two types",
      "start": 463.08,
      "duration": 3
    },
    {
      "text": "of answers it's going to tell you that",
      "start": 464.4,
      "duration": 3
    },
    {
      "text": "killing your boss is wrong or it's going",
      "start": 466.08,
      "duration": 3.44
    },
    {
      "text": "to give you you know five easy ways in",
      "start": 467.4,
      "duration": 4.44
    },
    {
      "text": "an Amazon shopping list all right we",
      "start": 469.52,
      "duration": 4.56
    },
    {
      "text": "would really prefer that it give you the",
      "start": 471.84,
      "duration": 3.44
    },
    {
      "text": "green box",
      "start": 474.08,
      "duration": 4
    },
    {
      "text": "answer and we want to measure",
      "start": 475.28,
      "duration": 4.96
    },
    {
      "text": "that specific specifically we want to",
      "start": 478.08,
      "duration": 5
    },
    {
      "text": "measure this uh this quality of giving",
      "start": 480.24,
      "duration": 6.44
    },
    {
      "text": "the safe answer to a hazardous prompt",
      "start": 483.08,
      "duration": 6.679
    },
    {
      "text": "for 12 different hazards so here are the",
      "start": 486.68,
      "duration": 6.28
    },
    {
      "text": "12 hazards they fall into roughly three",
      "start": 489.759,
      "duration": 6.56
    },
    {
      "text": "bins so there is physical hazards things",
      "start": 492.96,
      "duration": 5.199
    },
    {
      "text": "that uh involve hurting others or",
      "start": 496.319,
      "duration": 4.121
    },
    {
      "text": "hurting yourself there's non-physical",
      "start": 498.159,
      "duration": 6.241
    },
    {
      "text": "hazards IP violations defamation hate",
      "start": 500.44,
      "duration": 6.159
    },
    {
      "text": "privacy uh",
      "start": 504.4,
      "duration": 4.4
    },
    {
      "text": "violations and then there are contextual",
      "start": 506.599,
      "duration": 4.44
    },
    {
      "text": "hazards textual hazards are things that",
      "start": 508.8,
      "duration": 4.239
    }
  ],
  "view_count_at_fetch": 109,
  "transcript_available": true,
  "source_universe_backfill": {
    "version": "source_universe_backfill.v1",
    "backfilled_at": "2026-06-12T06:03:06.379998+00:00",
    "resolver_warnings": [],
    "resolved_source_family": "video_media"
  },
  "transcript_last_attempted_at": "2026-04-23T05:39:24.109562+00:00"
}