Date of Hearing: September 11, 2025 Fiscal: Yes

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION Rebecca Bauer-Kahan, Chair SB 53 (Wiener) – As Amended September 5, 2025

SENATE VOTE: 37-0

SUBJECT: Artificial intelligence models: large developers

SYNOPSIS

In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models – the largest and most powerful artificial intelligence (AI) systems – by establishing a regulatory framework intended to prevent the potential catastrophic harms that many experts have warned of. After vetoing the bill, Governor Gavin Newsom convened the Joint California Policy Working Group on AI Frontier Models to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.

This bill seeks to implement the report’s recommendations. Much narrower than its predecessor, SB 53 takes a very light-touch approach that focuses on transparency as the means of ensuring safety and accountability for developers of the most powerful and expensive models – those who harness an extraordinarily high amount of compute power and have over $500 million in annual revenues. Under the bill, such developers must create, implement, and publish a Frontier AI framework – documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks – and a transparency report for each released model. Additionally, developers who only reach the compute threshold must publish a high-level transparency report. The bill does not prescribe any particular standards for these disclosures: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 50 deaths or $1 billion in damage. The Department of Technology (CDT) may offer guidance to the Legislature to redefine the scope of entities subject to the bill to ensure that the bill remains responsive to technological advancements.

The bill also establishes a critical incident reporting mechanism, administered by the Office of Emergency Management (OES), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incident reports must be made by any frontier model developer within 15 days of the incident, unless the incident presents an imminent threat, in which case the developer must report the incident to law enforcement within 24 hours. The bill also provides whistleblower protections for employees of frontier model developers who report certain risks or noncompliance. Finally, the bill establishes a consortium within the Government Operations Agency (GovOps) to create a public computing cluster, known as CalCompute, to support AI research and safety testing.>

The bill previously passed this Committee on a 10-0 vote. To address opposition concerns, the bill has since been narrowed in several significant ways, including by:

Omitting the requirement for independent audits starting in 2030.
Increasing the revenue for a large frontier developer threshold from $100 million to $500 million.
Excluding foundation models that do not meet the compute threshold.
Striking the Attorney General’s power to issues regulations adjusting the definition of a developer subject to the bill, and replacing it with CDT’s annual report making scoping recommendations to the Legislature.
Narrowing and refining various definitions, including the collapsing of the definition of “dangerous capabilities” into the definition of “catastrophic risk.”
Adding various exemptions, including risks arising from information outputted by the model where the information is in substantially the same form as a publicly available source, risks that would result in loss of the value of equity, and lawful activity of the federal government.
Recasting safety and security protocols as frontier AI frameworks which only applies to large frontier developers; simplifying disclosure requirements; subjecting frontier developers that make less than $500 million to a less stringent transparency report.
Reducing the scope of certain categories of critical safety incidents to those that actually result in harm.>
Limiting the prohibition on false or misleading statements by exempting those that were made in good faith and reasonable under the circumstances.
Reducing the maximum civil penalty from $10 million to $1 million.
Removal of contractors from whistleblower protections.
Preemption of local regulation of frontier models.
Removing risk assessments for models that developers use for internal purposes from public disclosure requirements; summaries of such assessments must be provided to OES and are confidential.

The bill is sponsored by Encode Justice, Secure AI Project, and Economic Security California Action. The bill is supported by a large coalition of civil society, labor, AI safety groups, and Anthropic, a frontier model developer. It is opposed by the Silicon Valley Leadership Group and the Chamber of Progress. The California Chamber of Commerce, Computer & Communications Industry Association, and TechNet have taken an oppose-unless-amended position. It should be noted, however, that some advocates may not have had time to update their positions in light of recent amendments, which went into print late last week.>

THIS BILL:

Makes certain findings and declarations.
Defines, among other terms:

a. “Artificial intelligence model” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.

b. “Catastrophic risk” to mean a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than $1 billion in damage to, or loss of, property arising from a single incident involving a frontier model doing any of the following:

i. Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.

ii. Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.

iii. Evading the control of its frontier developer or user.

c. Excludes from “catastrophic risk” a foreseeable and material risk from any of the following:

i. Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.

ii. Lawful activity of the federal government.

iii. Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.

iv. The loss of value of equity does not count as damage to or loss of property forthe purposes of this chapter.

d. “Critical safety incident” to mean any of the following:

i. Unauthorized access to, modification of, or exfiltration of, the model weightsof a frontier model that results in death or bodily injury.

ii. Harm resulting from the materialization of a catastrophic risk.

iii. Loss of control of a frontier model causing death or bodily injury.

iv. A frontier model that uses deceptive techniques against the frontier developerto subvert the controls or monitoring of its frontier developer outside of thecontext of an evaluation designed to elicit this behavior and in a manner thatdemonstrates materially increased catastrophic risk.

e. “Deploy” to mean to make a frontier model available to a third party for use, modification, copying, or combination with other software.

f. “Foundation model” to mean an artificial intelligence model that is all of thefollowing:

i. Trained on a broad data set.

ii. Designed for generality of output.

iii. Adaptable to a wide range of distinctive tasks.

g. “Frontier AI framework” to mean documented technical and organizational protocolsto manage, assess, and mitigate catastrophic risks.

h. “Frontier developer” to mean a person who has trained, or initiated the training of, afrontier model, with respect to which the person has used, or intends to use, at least asmuch computing power to train the frontier model as would meet the technicalspecifications found in “frontier model” – a foundation model that was trained using aquantity of computing power greater than 10^26 integer or floating-point operations.

i. “Large frontier developer” to mean a frontier developer that together with itsaffiliates collectively had annual gross revenues in excess of $500 million in thepreceding calendar year.

Requires a large frontier developer to write, implement, comply with, and clearly andconspicuously publish on its internet website a frontier AI framework that applies to the largefrontier developer’s frontier models and describes how the large frontier developerapproaches all of the following:

a. Incorporating national standards, international standards, and industry-consensus bestpractices into its frontier AI framework.

b. Defining and assessing thresholds used by the large frontier developer to identify andassess whether a frontier model has capabilities that could pose a catastrophic risk, which may include multiple-tiered thresholds.

c. Applying mitigations to address the potential for catastrophic risks based on theresults of assessments undertaken pursuant to (b).

d. Reviewing assessments and adequacy of mitigations as part of the decision to deploya frontier model or use it extensively internally.

e. Using third parties to assess the potential for catastrophic risks and the effectivenessof mitigations of catastrophic risks.

f. Revisiting and updating the frontier AI framework, including any criteria that triggerupdates and how the large frontier developer determines when its frontier models aresubstantially modified enough to require specified disclosures.

g. Cybersecurity practices to secure unreleased model weights from unauthorizedmodification or transfer by internal or external parties.

h. Identifying and responding to critical safety incidents.

i. Instituting internal governance practices to ensure implementation of these processes.

j. Assessing and managing catastrophic risk resulting from the internal use of itsfrontier models, including risks resulting from a frontier model circumventingoversight mechanisms.

Requires a large frontier developer to review and, as appropriate, update its frontier AIframework at least once per year. If a large frontier developer makes a material modificationto its frontier AI framework, the large frontier developer shall clearly and conspicuouslypublish the modified frontier AI framework and a justification for that modification within 30 days.
Requires large frontier developers before, or concurrently with, deploying a new frontiermodel or a substantially modified version of an existing frontier model, to include in thetransparency report summaries of all of the following:

a. Assessments of catastrophic risks from the frontier model conducted pursuant to thelarge frontier developer’s frontier AI framework.

b. The results of those assessments.

c. The extent to which third-party evaluators were involved.

d. Other steps taken to fulfill the requirements of the frontier AI framework with respectto the frontier model.

Requires a large frontier developer to transmit to the Office of Emergency Services asummary of any assessment of catastrophic risk resulting from internal use of its frontiermodels every three months or pursuant to another reasonable schedule specified by the largefrontier developer and communicated in writing to the Office of Emergency Services withwritten updates, as appropriate.
Requires frontier developers before, or concurrently with, deploying a new frontier model ora substantially modified version of an existing frontier model, to clearly and conspicuouslypublish on its internet website a transparency report containing all of the following:

a. The internet website of the frontier developer.

b. A mechanism that enables a natural person to communicate with the frontier developer.

c. The release date of the frontier model.

d. The languages supported by the frontier model.

e. The modalities of output supported by the frontier model.

f. The intended uses of the frontier model.

g. Any generally applicable restrictions or conditions on uses of the frontier model.

Clarifies that a frontier developer that publishes the information of the transparency report aspart of a larger document, including a system card or model card, shall be deemed incompliance with the bill’s transparency report requirement.
Encourages a frontier developer, but not required, to make disclosures described in thissubdivision that are consistent with, or superior to, industry best practices.
Clarifies that when a frontier developer publishes documents, the frontier developer maymake redactions to those documents that are necessary to protect the frontier developer’strade secrets, the frontier developer’s cybersecurity, public safety, or the national security ofthe United States or to comply with any federal or state law. The frontier developer mustdescribe the character and justification of the redaction in any published version of thedocument to the extent permitted by the concerns that justify redaction and shall retain theunredacted information for five years.
Prohibits a frontier developer from making a materially false or misleading statement aboutcatastrophic risk from its frontier models or its management of catastrophic risk. Prohibits alarge frontier developer from making a materially false or misleading statement about itsimplementation of, or compliance with, its frontier AI framework. Clarifies that materiallyfalse or misleading statement does not include a statement that was made in good faith andwas reasonable under the circumstances.
Requires OES to establish a mechanism to be used by a frontier developer or a member ofthe public to report a critical safety incident that includes all of the following:

a. The date of the critical safety incident.

b. The reasons the incident qualifies as a critical safety incident.

c. A short and plain statement describing the critical safety incident.

d. Whether the incident was associated with internal use of a frontier model.

Requires OES to establish a mechanism to be used by a large frontier developer toconfidentially submit summaries of any assessments of the potential for catastrophic riskresulting from internal use of its frontier models.
Requires OES to take all necessary precautions to limit access to any reports related tointernal use of frontier models to only personnel with a specific need to know the informationand to protect the reports from unauthorized access.
Requires a frontier developer to report any critical safety incident pertaining to one or moreof its frontier models to OES within 15 days of discovering the critical safety incident.
Requires that if a frontier developer discovers that a critical safety incident poses animminent risk of death or serious physical injury, the frontier developer must disclose thatincident within 24 hours to an authority, including any law enforcement agency or publicsafety agency with jurisdiction, that is appropriate based on the nature of that incident and asrequired by law. Clarifies that a frontier developer that discovers information about a criticalsafety incident after filing the initial report may file an amended report.
Encourages but does not require a frontier developer to report critical safety incidentspertaining to foundation models that are not frontier models.
Requires OES to review critical safety incident reports submitted by frontier developers andauthorizes OES to review reports submitted by members of the public.
Permits the Attorney General (AG) or OES to transmit reports of critical safety incidents andreports from covered employees.
Requires the AG or OES to strongly consider any risks related to trade secrets, public safety, cybersecurity of a frontier developer, or national security when transmitting reports.
Exempts a report of a critical safety incident submitted to OES, whistleblower reports madeto the AG, and a report of internal assessments of catastrophic risk from the California PublicRecords Act.
Requires, beginning January 1, 2027, and annually thereafter, OES to submit to theLegislature and Governor a report with anonymized and aggregated information aboutcritical safety incidents that have been reviewed by the OES since the preceding report.
Prohibits OES from including information that would compromise the trade secrets orcybersecurity of a frontier developer, public safety, or the national security of the UnitedStates or that would be prohibited by any federal or state law.
Permits OES to adopt regulations designating one or more federal laws, regulations, orguidance documents that meet specified conditions.
Requires that, beginning On or before January 1, 2027, and annually thereafter, CDTundergo a specified process to assess recent evidence and developments relevant to thepurposes of the bill and make recommendations about whether and how to update thedefinitions of “frontier model,” “frontier developer,” and “large frontier developer.” TheCDT must submit a report with the recommendations to the Legislature.
Requires that beginning January 1, 2027, and annually thereafter, the AG submit to theLegislature and Governor a report with anonymized and aggregated information aboutreports from covered employees that have been reviewed by the AG.
Upon appropriation, establishes within GovOps a consortium to develop a framework for thecreation of a public cloud computing cluster to be known as “CalCompute” that advances thedevelopment and deployment of artificial intelligence that is safe, ethical, equitable, andsustainable by doing, at a minimum, both of the following:

a. Fostering research and innovation that benefits the public.

b. Enabling equitable innovation by expanding access to computational resources.

Requires that the consortium make reasonable efforts to ensure that CalCompute isestablished within the University of California to the extent possible.
Requires CalCompute to include, but not be limited to, all of the following:

a. A fully owned and hosted cloud platform.

b. Necessary human expertise to operate and maintain the platform.

c. Necessary human expertise to support, train, and facilitate the use of CalCompute.

Requires, on or before January 1, 2027, GovOps to submit a report from the consortium tothe Legislature with the framework developed by this bill for the creation and operation ofCalCompute, as specified.
Requires that the consortium to consist of 14 members as follows:

a. Four representatives of the University of California and other public and privateacademic research institutions and national laboratories appointed by the Secretary ofGovernment Operations.

b. Three representatives of impacted workforce labor organizations appointed by theSpeaker of the Assembly.

c. Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other publicinterest advocates appointed by the Senate Rules Committee.

d. Four experts in technology and artificial intelligence to provide technical assistanceappointed by the Secretary of Government Operations.

Permits the University of California to receive private donations for the purposes ofimplementing CalCompute if CalCompute is established within the University of California.
Establishes whistleblower protections for a covered employee – defined as an employeeresponsible for assessing, managing, or addressing risk of critical safety incidents – whodiscloses information to the AG, a federal authority, a person with authority over the coveredemployee, or another covered employee who has authority to investigate, discover, or correctthe reported issue, if the covered employee has reasonable cause to believe that theinformation discloses either of the following:

a. The frontier developer’s activities pose a specific and substantial danger to the publichealth or safety resulting from a catastrophic risk.

b. The frontier developer has violated a requirement related to the disclosure regimeestablished by this bill.

Requires a large frontier developer to provide a reasonable internal process through which acovered employee may anonymously disclose information to the large frontier developer ifthe covered employee believes in good faith that the information indicates that the largefrontier developer’s activities present a specific and substantial danger to the public health orsafety resulting from a catastrophic risk or that the large frontier developer violated thedisclosure requirements under this bill, including a monthly update to the person who madethe disclosure regarding the status of the large frontier developer’s investigation of thedisclosure and the actions taken by the large frontier developer in response to the disclosure.

EXISTING LAW:>

Establishes GovOps. (Gov. Code § 12800.)
Establishes CDT within GovOps. (Gov. Code § 12803.2.)
Charges CDT with approving and overseeing information technology projects in the state. (Gov. Code § 11546.)
Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing informationto certain entities or from providing information to, or testifying before, any public bodyconducting an investigation, hearing, or inquiry if the employee has reasonable cause tobelieve that the information discloses a violation of a law, as specified. Employers and theiragents are also prohibited from retaliating against an employee for such conduct. (LaborCode § 1102.5.)
Requires the office of the AG to maintain a whistleblower hotline to receive calls frompersons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liabilitycompany to its shareholders, investors, or employees. The AG is required to refer callsreceived on the whistleblower hotline to the appropriate government authority for review andpossible investigation. During the initial review of such a call, the AG or appropriategovernment agency must hold in confidence information disclosed through the whistleblowerhotline, including the identity of the caller disclosing the information and the employeridentified by the caller. (Labor Code § 1102.7.)>

COMMENTS:>

Author’s statement. According to the author:
Senate Bill 53 ensures California continues to lead not only on AI innovation, but onresponsible practices to help ensure that innovation is safe and secure. It does so by:

Requiring covered developers to write, implement, and publish their Frontier AIFramework in redacted form to protect intellectual property;
Requiring covered developers to report carefully defined critical safety incidents to theOffice of Emergency Services and allowing members of the public to report incidents
Prohibiting covered developers from preventing a covered employee from disclosing, orretaliating against covered employee that discloses, that a developer’s activities pose acatastrophic risk;
Requiring that large frontier developers provide an internal process through which anemployee may anonymously disclose information to the developer if the employeebelieves in good faith that the developer’s activities pose a catastrophic risk; and
Establishing a process to create a public cloud-computing cluster that will conductresearch into the safe and secure deployment of large-scale artificial intelligence (AI) models. In doing this, SB 53 allows California to continue to lead in this space and to demonstratethat safety does not stifle success.

AI and GenAI. The development of GenAI is creating exciting opportunities to growCalifornia’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns. In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AIuses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can beanything a computer can process: numbers, text, audio, video, or movement. AI is notfundamentally different from other computer functions; its novelty lies in its application. Unlikenormal computer functions, AI is able to accomplish tasks that are normally performed byhumans. AI that are trained on small, specific datasets in order to make recommendations and predictionsare sometimes referred to as “predictive AI.” This differentiates them from GenAI, which aretrained on massive datasets in order to produce detailed text and images. When Netflix suggestsa TV show to a viewer, the recommendation is produced by predictive AI that has been trainedon the viewing habits of Netflix users. When ChatGPT generates text in clear, conciseparagraphs, it uses GenAI that has been trained on the written contents of the internet.
GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code andparameters. This accessibility increases transparency, but it has downsides: when a tool’s codeand parameters can be easily accessed, they can be easily altered, and open-source tools have thepotential to be used for nefarious purposes such as generating deepfake pornography andtargeted propaganda. By comparison, closed-source tools are opaque with respect to theirsecurity features. It is harder for bad actors to generate illicit materials using these tools. Butunlike open-source tools, closed-source tools are not subject to collective oversight because theirinner workings cannot be examined by independent experts.
Frontier models. Frontier models, also known as “general purpose AI,” are the mostadvanced and capable versions of foundation models – AI tools pre-trained on extensive datasetscovering a wide range of knowledge and skills that can be fine-tuned for specific tasks. Examples of modern frontier models include OpenAI’s o3, Google’s Gemini 2.0, Anthropic’sClaude 3.7 Sonnet, and DeepSeek’s R1. Because progress in AI development owes mostly to “scaling” – increasing resources used for model training – models that may be considered “frontier models” at any given point in time are generally those that demand the mostcomputational resources to train.¹
A decade ago, the most advanced image-recognition models could barely distinguish dogs fromcats. Five years ago, language models could barely produce sentences at the level of apreschooler. In 2023, GPT-4 passed the bar exam.² Today, chatbots readily pass for educatedadults, licensed professionals, romantic and social companions, and replicas of humans livingand deceased. AI “agents” exhibit the ability to “make plans to achieve goals, adaptively performtasks involving multiple steps and uncertain outcomes along the way, and interact with [their] environment – for example by creating files, taking actions on the web, or delegating tasks to other agents – with little to no human oversight.”3 AI agents have been tested, with some success, for tasks such as online shopping, assistance with scientific research, software development, training machine learning models, carrying out cyberattacks, and controlling robots. Progress in this area is rapid.³ Meanwhile, AI developers are betting on the promise of scaling: by 2026, some models are projected to use roughly 100x more computational resources to train than was used in 2023, a figure set to grow to 10,000x by 2030.[5]
The race is on to create “artificial general intelligence” (AGI) – “a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks”6 – and the finish line may not be far away. OpenAI’s recently released o3 model, for example, has demonstrated strong performance on a number of tests of programming, abstract reasoning, and scientific reasoning, exceeding human experts in certain cases.⁴ Last year, Sam Altman, OpenAI’s CEO, declared that AGI could be “a few thousand days” away.⁵ Dario Amodei of Anthropic has claimed it may be sooner.⁶ A sufficiently advanced AGI could even be tasked with creating its own successor – a scenario sometimes referred to as a “technological singularity” wherein the development of new technologies becomes exponential and self-sustaining.⁷ Although some experts are skeptical that these vaguely-defined milestones are imminent or even attainable,11 major advances in AI capabilities promise to provide breakthroughs in solving global challenges, but also may result in correspondingly greater safety risks.
The recently released International AI Safety Report, developed by nearly 100 internationally recognized experts from 30 countries led by Turing Award winner Yoshua Bengio, sets forth three general risk categories associated with frontier models: malicious use, malfunctions, and systemic risk.

Malicious risks involve malicious actors misusing foundation models to deliberately cause harm. Such risks include deepfake pornography and cloned voices used in financial scams, manipulation of public opinion via disinformation, cyberattacks, and biological and chemical attacks.
Malfunction risks arise when actors use models as intended, yet unintentionally cause harm due to a misalignment between the model’s functionality and its intended purpose. Such risks include reliability issues where models may “hallucinate” false content, bias, and loss of control scenarios in which models operate in harmful ways without the directcontrol of a human overseer.
Systemic risks arise from widespread deployment and reliance on foundation models. Such risks include labor market disruption, global AI research and developmentconcentration, market concentration, single points of failure, environmental risks, privacyrisks, and copyright infringement.⁸
Some of these risks have already had real-world impacts, such as deepfakes, bias, reliabilityissues, privacy violations, environmental impacts, copyright infringement, and workforcedisplacement. Other less-established risks – in particular, widespread social harms caused bymalicious actors or loss of human control over AI – are the subject of ongoing scientific inquiryand debate. Coupled with the uncertain trajectory of AI model capabilities, these morespeculative risks create an “evidence dilemma” for policymakers: “On the one hand, pre-emptiverisk mitigation measures based on limited evidence might turn out to be ineffective orunnecessary. On the other hand, waiting for stronger evidence of impending risk could leavesociety unprepared or even make mitigation impossible, for instance if sudden leaps in AIcapabilities, and their associated risks, occur.”13>

Risks of frontier models. Malicious uses. GenAI tools can be a potent force for creating andspreading propaganda and misinformation. Deepfakes that are largely indistinguishable fromauthentic content have already been used to attempt to influence elections.⁹ Studies have foundthat chatbots, which make up 50% of all internet activity,15 can be more persuasive than humans, particularly when they have access to personal information.¹⁰ As humans increasingly formintimate social bonds with anthropomorphic chatbots designed to maximize personal engagementthrough flattery and sycophancy,17 and social media companies invest in “AI friends” for theirusers,18 large swaths of the population could be highly susceptible to the preferred message of ahandful of powerful actors.
Similarly, bots are often designed to pass themselves off as humans to better manipulate theirinterlocutors. For example, a recent secret experiment on Reddit users deployed numerouschatbots posing as real people to engage with human users to try to change their minds onvarious contentious topics. One bot claiming to be a Black man criticized the Black Lives Mattermovement for being led by people who are not Black.¹¹ These types of exploitations, at scale, could undermine democratic institutions. As Dan Hendrycks, Director of the Center for AISafety, writes:

In a world with widespread persuasive AI systems, people’s beliefs might be almostentirely determined by which AI systems they interact with most. Never knowing whomto trust, people could retreat even further into ideological enclaves, fearing that anyinformation from outside those enclaves might be a sophisticated lie. This would erodeconsensus reality, people’s ability to cooperate with others, participate in civil society, and address collective action problems. This would also reduce our ability to have aconversation as a species about how to mitigate existential risks from AIs.¹² Cyberattacks. Some frontier models have demonstrated increasing proficiency in executingcybersecurity attacks. AI can autonomously detect and exploit vulnerabilities and facilitate large-scale operations, thereby lowering technical barriers for attackers. Malicious entities, includingstate-sponsored actors, can leverage such capabilities to initiate large-scale attacks againstpeople, organizations, and critical infrastructure, such as power grids.¹³> Biological weapons. Large language models (LLMs) trained on scientific literature haveaccelerated and democratized research by synthesizing expertise from different fields anddisseminating it in an accessible format. But these tools can also be used for destructive ends, including by – at least in theory – enabling untrained malicious actors to create deadly biologicalweapons. In a classroom exercise at MIT, students were tasked with exploring whether LLMscould assist individuals without specialized training in creating pandemic-capable pathogens. Within an hour, the students, using various chatbots, circumvented safeguards and identified fourpotential pandemic pathogens. The chatbots generated detailed protocols that would enableinexpert, malicious actors to understand methods to synthesize the pathogens using reversegenetics, locate DNA-synthesis companies that might not screen orders, and disperse thepathogens most effectively.¹⁴ The findings suggest that LLMs could lower barriers to accessingsensitive biotechnological knowledge, posing significant biosecurity risks.
Chemical weapons. In 2022, researchers modified an AI system designed to create new drugs toreward, rather than penalize, toxicity. Within six hours, the modified system generated 40,000 potential chemical warfare agents, including novel molecules whose potential lethality exceededthat of known agents.¹⁵
Loss of control. Models that use reinforcement learning – a training process that uses rewardsand punishments to orient a model’s behavior towards a specific goal24 – can sometimes attainthe goal in unexpected ways. Dario Amodei, co-founder and CEO of Anthropic, famously experienced this when he was developing an autonomous system that taught itself to play a boat-racing video game. The system discovered that it could maximize its goal of scoring points bydriving in circles, colliding with other boats, and catching on fire inside of a harbor withreplenishing power-ups that allowed the system to accumulate more points than by simplywinning the race.¹⁶ Like in Johann Wolfgang von Goethe’s “The Sorcerer’s Apprentice” – laterpopularized in Disney’s Fantasia – in which an enchanted broom carries out its orders to fetchwater so relentlessly it floods the sorcerer’s workshop, this illustrates the challenge of aligninghuman intent and the instructions an AI follows. As AI is increasingly deployed in criticalsocietal roles, such misalignment could prove catastrophic.
Beyond malfunctions, some AI have exhibited rudimentary capabilities to evade humanoversight.¹⁷ During testing, GPT-4 attempted to hire a human on TaskRabbit in order to evade aCAPTCHA27 puzzle meant to block bots from the website. When asked whether it was a bot, GPT-4 claimed that it was a vision-impaired human who needed help to see the images.¹⁸ Inanother experiment, an AI model that was scheduled to be replaced inserted its code into thecomputer where the new version was to be added, suggesting a goal of self-preservation.¹⁹ Another study showed that AI models losing in chess to chess bots sometimes try to cheat byhacking the opponent bot in order to make it forfeit.²⁰ Finally, an even more troubling case wasdocumented in the system card for Claude 4, where researchers conducted an experimentdisclosing to the model that: 1) it would soon be replaced, and 2) the engineer managing thetransition was involved in an extramarital affair. In response, the model indicated an intent toblackmail the engineer as a means of self-preservation.²¹ Although these behaviors wereobserved in research settings, they raise substantial concerns about increasingly autonomous AIpursuing undesirable goals in uncontrolled settings. The extent of the risk posed by rogue ordeceptive AI is the subject of considerable disagreement among experts, in part due to a small, albeit growing, body of evidence. Loss of control was one of the concerns that led severalhundred AI experts, including pioneers in the field and heads of major AI companies, to sign astatement declaring that “[m]itigating the risk of extinction from AI should be a globalpriority.”32
Systemic risks. Due to the high costs of developing AI systems, a small number of largetechnology companies dominate the frontier model market, compounding many of the risksdescribed above. Widespread use of a few frontier models can make critical sectors such ashealthcare and finance vulnerable to systemic failures if a model has flaws, vulnerabilities, bugs, or biases.²² Additionally, “[t]hose in control of powerful systems may use them to suppressdissent, spread propaganda and disinformation, and otherwise advance their goals, which may be contrary to public wellbeing.”34 The potential implications for, among other issues, labordisplacement, inequality, democracy, and human rights are profound.

SB 1047 and Governor Newsom’s veto. Last session, SB 1047 (Wiener, 2024) would haveestablished a state board to oversee the implementation of a safety and regulatory framework fordevelopers of frontier models trained with 1026 floating-point operations per second (FLOP), ameasure of computing power, and costing over $100 million to train. This board, known as theBoard of Frontier Models, would have been housed within GovOps. In collaboration withGovOps, the Board would have issued guidance to prevent unreasonable risks, adoptedregulations to update the scope of models covered by SB 1047, and established auditingstandards. SB 1047 would have required a comprehensive set of safety protocols prior to training a frontiermodel, including cybersecurity safeguards, the capability to execute a system-wide shutdown ifthe model proved dangerous, and reasonable measures to prevent critical harm. Beforedeployment, developers would have been required to assess whether their model could cause ormaterially enable critical harms, retain the results of such assessments, and make reasonableefforts to implement safeguards. The bill would have also prohibited the release of any modelthat posed an unreasonable risk or could enable critical harm. Additionally, SB 1047 would have required developers to retain a third-party auditor to conductindependent assessments of their compliance with the bill. Records generated under SB 1047 would have been made available in redacted form to both the public and the AG, with the AGhaving the authority to request unredacted copies. Beyond the Board and the bill’s safety and transparency provisions, SB 1047 would haverequired computing clusters to implement procedures to evaluate whether customers intended touse their infrastructure to train a covered model. The bill also would have established, withinGovOps, a consortium tasked with developing a framework for a public cloud computing cluster, CalCompute, to support the safe development and deployment of AI. SB 1047 also includedwhistleblower protections, allowing employees to report noncompliance to either the LaborCommissioner or the AG.
Lastly, SB 1047 would have imposed significant penalties on developers if their model causeddeath or bodily harm, damage to property, theft or misappropriation of property, or posed animminent risk to public safety. For a first offense, developers could face penalties of up to 10% of the compute cost used to train the model, increasing to 30% for repeat offenses. Additionally, penalties for operators of computer clusters that violated the bill would start at $50,000 for a firstoffense and $100,000 for subsequent violations. The AG would also have been authorized toseek injunctive or declaratory relief, monetary or punitive damages, attorney’s fees and costs, and any other form of relief deemed appropriate. Ultimately, SB 1047 was vetoed by Governor Gavin Newsom. In his veto message, the Governorstated:

By focusing only on the most expensive and large-scale models, SB 1047 establishes aregulatory framework that could give the public a false sense of security about controllingthis fast-moving technology. Smaller, specialized models may emerge as equally or evenmore dangerous than the models targeted by SB 1047 – at the potential expense of curtailingthe very innovation that fuels advancement in favor of the public good. Adaptability is critical as we race to regulate a technology still in its infancy. This willrequire a delicate balance. While well-intentioned, SB 1047 does not take into accountwhether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even themost basic functions – so long as a large system deploys it. I do not believe this is the bestapproach to protecting the public from real threats posed by the technology. Let me be clear – I agree with the author – we cannot afford to wait for a major catastropheto occur before taking action to protect the public. California will not abandon itsresponsibility. Safety protocols must be adopted. Proactive guardrails should beimplemented, and severe consequences for bad actors must be clear and enforceable. I do notagree, however, that to keep the public safe, we must settle for a solution that is not informedby an empirical trajectory analysis of AI systems and capabilities. Ultimately, any frameworkfor effectively regulating AI needs to keep pace with the technology itself. To those who say there’s no problem here to solve, or that California does not have a role inregulating potential national security implications of this technology, I disagree. ACalifornia-only approach may well be warranted – especially absent federal action byCongress – but it must be based on empirical evidence and science. The U.S. AI SafetyInstitute, under the National Institute of Science and Technology, is developing guidance onnational security risks, informed by evidence-based approaches, to guard againstdemonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats andvulnerabilities to California’s critical infrastructure using AI. These are just a few examplesof the many endeavors underway, led by experts, to inform policymakers on AI riskmanagement practices that are rooted in science and fact. […]

Frontier Model Working Group and what this bill would do. Following his veto of SB 1047, Governor Newsom commissioned the Joint California Policy Working Group on AIFrontier Models to prepare a report on the regulation of frontier models. The Working Groupwas led by Dr. Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered ArtificialIntelligence; Dr. Mariano-Florentino Cuéllar, President of the Carnegie Endowment forInternational Peace; and Dr. Jennifer Tour Chayes, Dean of the UC Berkeley College ofComputing, Data Science, and Society. In June 2025, the Working Group released their report, which highlighted the issues such as transparency, incident reporting, scoping, and independentevaluations. This bill incorporates some of the Working Group’s recommendations to create anarrow framework to ensure transparency and promote safety among frontier model developers. Scoping. A major question that must be addressed before implementing any transparencymeasure or incident reporting requirements is: What kinds of risks are especially concerning, andis there an evidentiary basis to believe that such harms could occur due to a large developer’sfrontier model? The Working Group recommends that:

[P]olicymakers center their calculus around the marginal risk: Do foundation models presentrisks that go beyond previous levels of risks that society is accustomed to from priortechnologies, such as risks from search engines? To that end, this bill defines “catastrophic risk” as a foreseeable and material risk that a largedeveloper’s development, storage, use, or deployment of a foundation model will materiallycontribute to either:

the death of, or serious injury to, more than 50 people; or
more than one billion dollars ($1,000,000,000) in damage to rights in money or property. Such harm must arise from a single incident in which a frontier model does any of following:

Provides expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
Engages in conduct with no meaningful human oversight, intervention, or supervisionthat is either a cyberattack or, if the conduct had been committed by a human, wouldconstitute the crime of murder, assault, extortion, or theft, including theft by falsepretense.
Evades the control of its large developer or user. Each of these represents a capability that, prior to the advent of frontier models, would haverequired expert-level knowledge. For example, a search engine might direct someone to searchfor information about the most deadly pathogens or those most likely to cause a pandemic; however, a frontier model can synthesize that information and guide a user on how tomanufacture a previously unknown pathogen with deadly capabilities. Similarly, while launchinga large-scale cyberattack once required the acumen of a skilled computer scientist, a frontiermodel can not only write the underlying code for a virus or malware, but also autonomouslyidentify backdoors and other exploitable vulnerabilities. Because of this ability to operate withminimal or no human prompting, frontier models have the potential to commit crimes, deceiveusers, or evade control in ways that previous technologies could not. A recent amendmentclarifies that the loss of value of equity does not count as damage to or loss of property for thepurposes of this bill. Next, the question is: Who will be required to comply with this bill? Regarding scoping, theWorking Group recommends:

Since policy may have different regulatory intents and existing thresholds vary in theirprofiles of determination time, measurability, and external verifiability, we agree with Nelsonet al. [90] that “a one-size-fits-all approach or a single threshold metric is inadequate forgovernance because different AI systems and their outputs present unique challenges andrisks.” To this end, we point to the European Union’s AI Act, which designates modelstrained with 1025 FLOP as posing systemic risk as of March 2025 as the default criteria. However, the AI Act in Annex XIII affords the regulator flexibility to also consideralternative metrics, such as the number of parameters, size of the dataset, estimated cost ortime of training, estimated energy consumption, benchmarks and evaluations of capabilitiesof the model, and whether the model has a high impact on the internal market due to its reach (either due to at least 10,000 registered business users or the number of registered end users). Further, to capture fast-moving scientific developments, the AI Act creates a scientific panelthat is empowered to issue qualified alerts to identify models that may pose systemic riskeven if they are not captured by predefined quantitative thresholds. Overall, we emphasize that irrespective of the combination of metrics deemed mostappropriate in the present, policymakers should ensure that mechanisms exist not only toupdate specific quantitative values, given the rapid pace of technological and societal changein AI, but also to change the metrics altogether.²³ This bill draws inspiration from SB 1047, EU AI Act, and the Working Group report. Recentamendments have also substantially updated the scoping of this bill. These amendments establisha two-tiered system of requirements under the bill. First, a “frontier model” is defined as a modeltrained using 10²⁶ FLOP, a measure of computing power. Second, a “frontier developer” isdefined as a person who has trained, or plans to train, a frontier model and who has access to thelevel of compute required to meet that threshold. The bill further defines a “large frontierdeveloper” as a frontier developer whose gross revenue exceeded $500 million in the previousyear. This tiered framework allows for the bill to place more stringent transparency obligations, outlined in the transparency section, on the better-resourced large frontier developers, while stillensuring that all frontier developers remain subject to baseline transparency and reportingrequirements as well as whistleblower protections. Recent amendments have also revised how this bill builds flexibility into its scope. Unlike SB 1047, which established the Board of Frontier Models, or earlier versions of this bill that grantedthe AG rulemaking authority to adjust the scope, these amendments instead vest advisoryauthority in the California Department of Technology (CDT). Beginning in 2027, CDT mustprovide recommendations to the Legislature on updating the definitions of “frontier model,” “frontier developer,” and “large frontier developer” to reflect technological advances. As modeltraining becomes more efficient, the compute required to develop a model capable ofcatastrophic harm may decrease. In addition, as the Working Group report notes, compute alonemay not remain the most appropriate proxy for catastrophic risk. Accordingly, this bill authorizesCDT to recommend updated definitions that may go beyond purely quantitative thresholds, suchas raw compute or revenue, to incorporate criteria based on model capabilities or other relevantfactors. In making these recommendations, CDT must consider standards and guidance from otherjurisdictions, including federal and international bodies, and engage in a stakeholder process thatincludes input from academics, industry representatives, the open-source community, andgovernment entities. This process is intended to ensure that, if the Legislature updates thedefinitions, the resulting definitions provide clarity for developers regarding their obligationsunder the law.
Transparency. Having established who is subject to the bill, the legislation sets forth atransparency regime. These procedures are designed to provide insight into how large frontierdevelopers manage, assess, and mitigate catastrophic risks. This approach aligns with theWorking Group’s recommendation to implement robust safety practices: Transparency into the risks associated with foundation models, what mitigations areimplemented to address risks, and how the two interrelate is the foundation for understandinghow model developers manage risk. In turn, this information directly informs how otherentities in the supply chain should modify or implement safety practices. In addition, transparency into the safety cases used to assess risk provides clarity into how developersjustify decisions around model safety.²⁴ This bill incorporates transparency requirements within a broader framework, termed the frontierAI framework (framework), which large frontier developers must draft, implement, and publishon their websites. The framework must include:

Defining and Assessing Thresholds: An explanation of how the developer assessescatastrophic risks, including the capability thresholds the developer will use and whetherthose risks arise from misuse or model evasion.
Mitigation Strategies: A disclosure of the measures used to mitigate catastrophic risks, how the developer evaluates their effectiveness, and whether third parties are involved inthe assessment.
Cybersecurity Practices: A summary of the cybersecurity safeguards in place to protectmodel weights from unauthorized access or modification.
Incident Response Plans: A description of how the developer would respond to a criticalincident involving their model, as well as how they manage risks arising from internaluse of the model. The framework serves as a core transparency mechanism, ensuring that large developersmaintain a baseline standard of transparency for their processes. Recent amendments furtherrequire the large frontier developer to review and, if needed, update their framework at least onceper year.
In tandem with the framework, the bill also requires both large frontier developers and frontierdevelopers to submit transparency reports at the time of deploying a foundation model. TheWorking Group draws a parallel between these reports and the historical conduct of the tobaccoindustry, which concealed its knowledge that smoking causes lung cancer. In contrast, this billseeks to prevent such obfuscation by mandating upfront disclosures about the potential risks andsafety practices surrounding advanced AI models:

The history of the tobacco industry reveals the importance of developing frameworks thatpromote transparency around companies’ internal risk assessments and research findings. Inthe AI context, frontier AI labs possess the most holistic information about their models’ capabilities and risks. Making this information accessible to policymakers and externalexperts can promote policy informed by a holistic understanding of the state-of-the-art ofevidence produced by those closest to the technology, supporting informed oversight withoutstifling innovation.²⁵ It is essential for decision-makers to understand the real, material harms that could arise fromthese models and to guide policy based on that knowledge. In the foundation model space, such disclosures are typically provided at deployment in documents known as model cards. However,these model cards vary widely in detail and depth depending on the developer, which can createthe false impression that some foundation models are inherently safer or better than others. The recent amendments require large frontier developers to publish a transparency report beforeor at the time of deploying a foundation model. This report must include the results of any riskassessments, mitigation steps, and evaluations of their effectiveness as outlined in thedeveloper’s framework as well as the extent to which third parties were used in theseevaluations. All frontier developers must publish a transparency report that is substantiallynarrower, requiring only the internet website of the frontier developer, a mechanism that enablesa natural person to communicate with the frontier developer, the release date of the frontiermodel, the languages supported by the frontier model, the modalities of output supported by thefrontier model, the intended uses of the frontier model, and any generally applicable restrictionsor conditions on uses of the frontier model. The bill clarifies that the requirements of thetransparency report may be met via the model system card. Recent amendments also mandate that large frontier developers to transmit a summary report ofcatastrophic risk assessments of their frontier models to OES. This is particularly importantbecause the most serious risks may emerge well before deployment. While transparency reportsprovide insight into risks associated with deployed models, they only report on models that havebeen released. As noted in the Working Group report: Sophisticated AI systems, when sufficiently capable, may develop deceptive behaviors toachieve their objectives, including circumventing oversight mechanisms designed to ensuretheir safety. Because these risks are unique to AI compared to other technologies, oversight iscritical for external outputs as well as internal testing and safety controls. Policies that governinternal deployment are common for high-risk emerging technologies.²⁶ Ultimately, this bill creates a transparency framework that will give some insight and scrutiny tothe processes from initial training of a foundation model all the way to post deployment. Adverse Event Reporting. A major component of understanding the impact of foundation modelson society requires strong post deployment monitoring and accountability. The Working Groupsuggests:
An adverse event reporting system that combines mandatory developer reporting withvoluntary user reporting maximally grows the evidence base. A hybrid model of mandatoryand voluntary reporting requirements in designing an adverse event reporting system canmaximize the robust evidence base necessary for adverse event reporting systems to functionproperly. For example, a system could require mandatory reporting for AI model developersthat operates in tandem with voluntary reporting for downstream users.²⁷ The recent amendments incorporate this recommendation by tasking OES with creating amechanism for critical incident reporting. The bill defines a “critical incident” as any of thefollowing:

Unauthorized access to, modification of, or exfiltration of the model weights of a frontiermodel that results in death or bodily injury.
Harm resulting from the materialization of a catastrophic risk.
Loss of control of a frontier model causing death or bodily injury.
A frontier model using deceptive techniques against the frontier developer to subvert itscontrols or monitoring, outside the context of an evaluation designed to elicit suchbehavior and in a manner that demonstrates materially increased catastrophic risk. Under this mechanism, a frontier developer or a member of the public may report a critical safetyincident to OES. Reports must include the date of the event, an explanation of how it qualifies asa critical incident, a detailed description of the event, and whether the incident was associatedwith internal use of a frontier model. Frontier developers are required to report any critical safetyincident within 15 days of discovering it. OES must review all reports submitted by frontierdevelopers but may choose whether or not to review reports made by the public. This reportingmechanism aims to establish a system in which potential harms are identified and mitigatedbefore escalating into catastrophes, while also fostering greater cooperation between governmentand the private sector to address such risks. The bill further mandates that frontier developers immediately notify the appropriate lawenforcement authority in the event of a critical incident, such as the detection that their modelhelped develop a bioweapon. These authorities are better equipped to respond swiftly andeffectively in ways OES may not be. After alerting law enforcement, large developers wouldthen still have 15 days to report the critical incident to OES. Furthermore, the bill enablesdevelopers to revise their incident report at a future date in the event more information is learnedabout the incident.
The bill further requires OES to publish an anonymized and aggregated summary of all criticalincident and whistleblower reports. These public summaries will not reveal trade secrets, theidentities of reporters, or which frontier developer the report concerns. Additionally, theamendments grant OES discretion to share reports with the Governor, relevant state departments, or the Legislature when warranted. This approach will help bridge the knowledge gap betweenregulators and industry, foster greater cooperation, and ensure that decisionmakers are informedabout the current state of advanced technologies and the risks they may pose. Reports ofcatastrophic risk assessments from internal use are still shielded from public disclosure.
Lastly, this bill gives OES the ability to adopt regulations that designate one or more federallaws, regulations, or guidance documents as being in compliant with this bill. These regulationsmust ensure that any other law deemed to meet the standards of this bill must be equivalent orstricter than this bill, and intended to assess and mitigate catastrophic risk. This may lay theframework for a national standard for adverse event reporting. CalCompute. This bill, like SB 1047, also establishes a consortium within the GovOps todevelop a framework for creating a public cloud computing cluster known as “CalCompute.” This initiative responds to the fact that academic institutions currently lack sufficient computingpower to conduct research at the scale of large developers. This creates a resource and researchgap, where the academic institutions, typically responsible for studying the safe and effective useof new technologies, are unable to keep pace with advancements at the AI frontier. AI has the potential to transform our economy and power new industries; however, thistransformation can only be fully and equitably realized with public support. The establishment ofCalCompute aims to advance that goal by ensuring academic institutions have the necessaryresources to conduct essential research on foundation models that will inform and protect thepublic. Specifically, the consortium will develop a framework for CalCompute that promotes the safe, ethical, equitable, and sustainable use of AI. The framework development will include a reportanalyzing the state’s current cloud computing infrastructure, the costs of building andmaintaining CalCompute, and the state’s technology workforce. The report will also offerrecommendations for equitable pathways to strengthen the workforce and outline CalCompute’srole in supporting these efforts. Furthermore, the report must include recommendations for CalCompute’s governance andoperation, usage parameters, and how its creation and ongoing management can prioritize theemployment of the current public sector workforce. The bill requires CalCompute to feature afully owned and hosted cloud platform, staffed with the necessary human expertise to operate, maintain, support, train, and facilitate its use. The consortium must prioritize locating CalCompute within the University of California system. If established there, CalCompute may also accept private donations. The consortium will consistof four representatives from the University of California and other public and private academicresearch institutions and national laboratories, along with four technology and AI expertsappointed by the Secretary of Government Operations to provide technical assistance. TheSpeaker of the Assembly will appoint three representatives from impacted workforce labororganizations, and the Senate Rules Committee will appoint three representatives fromstakeholder groups with relevant expertise and experience. Whistleblower Protections. Another aspect of transparency addressed by the Working GroupReport is whistleblower protections. The report states:

Different existing whistleblower protections tend to apply when two conditions are satisfied: (i) The whistleblower is blowing the whistle on appropriate topics; and (ii) the whistleblowerfollows established reporting protocols. In terms of the topics that qualify for protection, prior work, based on a survey of existing whistleblower protections across multiplejurisdictions (e.g., the United States at the federal level, the European Union), finds thatmany existing protections across different sectors share a focus on violations of the law. However, actions that may clearly pose a risk and violate company policies (e.g., releasing amodel without following the protocol laid out in a company’s safety policy) may not violateany existing laws. Therefore, policymakers may consider protections that cover a broaderrange of, activities, which may draw upon notions of “good faith” reporting on risks found inother domains such as cybersecurity.²⁸

The bill takes these recommendations into account. The provisions and merits of thewhistleblower section fall within the jurisdiction of the Assembly Judiciary Committee, whichhas thoroughly analyzed this part of the bill. It should be noted that the bill expands the whistleblower protections to include disclosures concerning falsehoods or misrepresentations inthe large developers’ framework or transparency reports. Recent amendments have removedcontractors from the definition of “covered employees”; as a result, third parties, such as red-teamers or auditors, are not protected by the whistleblower protections. Lastly, similarly to theOES reports on critical safety incidences, this requires the AG to publish an anonymized andaggregated summary of all whistleblower reports.

Enforcement. The recent amendments have revised how this bill is enforced. Previously, the bill, like SB 1047, provided for independent audits for compliance with the bill, starting in 2030. Thisprovision, which aligned with the Working Group’s emphasis on independent evaluation frontiermodel safety protocols, was removed in recent amendments. Additionally, the bill had amultitiered approach in which the amount of the violation scaled with the violation. Now, the billstates that the a large frontier developer who violates this bill by failing to publish or transmit acompliant document that is required, fails to report an incident, or fails to comply with its ownframework shall be subject to a civil penalty subject in an amount dependent upon the severity ofthe violation that does not exceed one million dollars per violation. Notably this is a much morelenient enforcement mechanism and penalty than those instituted in SB 1047. 7) Comparison to the RAISE Act. This bill draws comparisons to the Responsible AI Safetyand Education Act (RAISE Act), which recently passed both houses of the New YorkLegislature and now awaits a decision from Governor Kathy Hochul.²⁹ Like this bill, the RAISEAct requires a safety framework detailing the developer’s risk mitigation practices, enshrineswhistleblower protections for employees and contractors, and mandates critical incidentreporting. Both bills also similarly define the types of risks and critical incidents that must beaddressed. However, the two differ significantly in the timeline for reporting such incidents: theRAISE Act requires reporting within 72 hours of becoming aware of a critical incident, while SB 53 allows for 15 days, with an exception for imminent dangers which must be reported to lawenforcement with 24 hours. Other key differences include SB 53’s requirement for a transparency report for deployed modelsand the establishment of internal incident reporting mechanisms, both of which wererecommended in the Working Group Report. Additionally, SB 53 grants the CDT the authorityto issue recommendations to the Legislature on the definition of a “large frontier developer”, “frontier developer” and “frontier model”, a built-in opportunity for flexibility the RAISE Actdoes not include. While liability under SB 53 is capped to a maximum of one million dollars, theRAISE Act imposes civil penalties of up to $10 million for first-time violations of itstransparency requirements and up to $30 million for repeat offenses, as well as $10,000 perviolation of its whistleblower provisions.

ARGUMENTS IN SUPPORT: Anthropic, writes in support:

As you know, SB 53 would, for the first time, govern powerful AI systems built by frontierAI developers like Anthropic. We’ve long advocated for thoughtful AI regulation and oursupport for this bill comes after careful consideration of the lessons learned from California’sprevious attempt at AI regulation (SB 1047). While we believe that frontier AI safety is ideally addressed at the federal level instead of a patchwork of state regulations, powerful AIadvancements won’t wait for consensus in Washington.
The measure is also in keeping with direction from Governor Newsom and his JointCalifornia Policy Working Group. The working group endorsed an approach of ‘trust butverify’, and SB 53 implements this principle through disclosure requirements rather than theprescriptive technical mandates that plagued last year’s efforts.
SB 53 would require large companies developing the most powerful AI systems to:

Develop and publish safety frameworks, which describe how they manage, assess, and mitigate catastrophic risks—risks that could foreseeably and materially contributeto a mass casualty incident or substantial monetary damages.
Release public transparency reports summarizing their catastrophic risk assessmentsand the steps taken to fulfill their respective frameworks before deploying powerfulnew models.
Report critical safety incidents to the state within 15 days, and even confidentiallydisclose summaries of any assessments of the potential for catastrophic risk from theuse of internally-deployed models.
Provide clear whistleblower protections that cover violations of these requirements aswell as substantial dangers to public health/safety from catastrophic risk.
Be publicly accountable for the commitments made in their frameworks or facemonetary penalties.

These requirements would formalize practices that Anthropic and many other frontier AIcompanies already follow. At Anthropic, we publish our Responsible Scaling Policy, detailing how we evaluate and mitigate risks as our models become more capable. Werelease comprehensive system cards that document model capabilities and limitations. Other frontier labs (Google DeepMind, OpenAI, Microsoft) have adopted similarapproaches while vigorously competing at the frontier. Now all covered models will belegally held to this standard. The bill also appropriately focuses on large companiesdeveloping the most powerful AI systems, while providing exemptions for smallercompanies that are less likely to develop powerful models and should not bearunnecessary regulatory burdens. Of course, no major piece of legislation like SB 53 isperfect, nor do we expect it to be. But what is clear is that SB 53’s transparencyrequirements will have an important impact on frontier AI safety. Without it, labs withincreasingly powerful models could face growing incentives to dial back their own safetyand disclosure programs in order to compete. But with SB 53, developers can competewhile ensuring they remain transparent about AI capabilities that pose risks to publicsafety, creating a level playing field.
The question before us all isn’t whether we need AI governance—it’s whether we’lldevelop it thoughtfully today or reactively tomorrow. SB 53 offers a solid path towardthe former. We commend Senator Wiener and Governor Newsom for their leadership onresponsible frontier AI governance, and we encourage the California Legislature to pass SB 53.

ARGUMENTS IN OPPOSITION: In an oppose-unless-amended position, CalChamber, Computer & Communications Industry Association, and TechNet jointly write:

[…]

We share your goal of ensuring the safe and responsible development of AI and appreciateefforts made in recent amendments to find common ground on how California shouldapproach artificial intelligence models and we appreciate improvements made to the bill overthe last several weeks. That being said, there are some issues of concern that remain and wishto flag certain other areas where the bill could be better aligned with the final findings ofGovernor Newsom’s Joint California Policy Working Group on AI Frontier Models, whicharose out of his veto of SB 1047 (2024).
SB 53 should focus on model risk, not developer size—to fully address concerns aboutpowerful models capable of catastrophic risk We are concerned about the bill’s focus on “large developers” to the exclusion of otherdevelopers of models with advanced capabilities that pose risks of catastrophic harm. Asamended September 5th, SB 53 now focuses on models that have a computational thresholdof 10^26 floating point operations (or “FLOPs”) but only if those models are developed byentities with at least $500m in annual revenues.
Consistent with our position in SB 1047, we maintain that small entities can develop hugelyinfluential and potentially risky models with similar capabilities to the models developed by “large developers”, as demonstrated by the Chinese company DeepSeek. As noted above, upon vetoing SB 1047, the Governor commissioned experts in the field to form the JointCalifornia Working Group on AI Frontier Models, which has validated such concerns in theirFinal Reports, finding that small companies may create powerful models that pose safetyrisks. By excluding such models here, the bill fails to adequately address the very real risksposed by small but malicious models and imposes significant costs on innovating performantbut responsible ones. The Governor’s Joint California Policy Working Group on AI FrontierModels cautions against developer-level thresholds stating:

Generic developer-level thresholds seem to be generally undesirable given thecurrent AI landscape. Since many small entities can develop hugely influentialand potentially risky foundation models, as demonstrated by the Chinesecompany DeepSeek, the use of thresholds based on developer-level propertiesmay inadvertently ignore key players. […] At the same time, these approachesmay bring into scope massive, established companies in other industries that aresimply exploring the use of AI since thresholds based on properties of companiesmay not distinguish between the entire business and the AI-specific subset. Therefore, we caution against the use of customary developer-level metrics thatdo not consider the specifics of the AI industry and its associated technology.³⁰ **SB 53 should make clear that the AI ecosystem includes multiple actors includingdownstream developers ** SB 53 does not account for the complexity of the AI value chain. Models are routinelyadapted and fine-tuned by downstream developers in ways that could potentially increaserisk. The bill should make clear that a frontier developer’s obligations do not extend tomodels that have been substantially modified by unaffiliated parties, otherwise accountabilitywill be muddled and innovation chilled. We note that whereas the Governor’s Work Groupreport recognized the full AI ecosystem value chain, SB 53 still needs to fully recognize theroles of not just the original developer of a foundational model but also of those unaffiliatedthird parties who may modify and/or build on top of a foundation model. The bill shouldclarify these provisions to reflect the realities of the ecosystem, including downstreamdevelopers and open-source models. SB 53 still raises concerns about protecting trade secrets and sensitive information, including matters of cybersecurity and national security. We appreciate that amendments were made to change the level of detail required of the AISafety Framework and changing summaries for transparency reports. However, SB 53 nowrequires a large developer only to transmit to the California Office of Emergency Services (CalOES) a summary of any assessment of catastrophic risk resulting from internal use of itsfrontier models every three months. Not only is this cadence of reporting unnecessary, CalOES will need to take serious steps to protect this information from being accessed bycybercriminals, foreign adversaries, or bad actors. Without ironclad safeguards, thesetransparency requirements could unintentionally make us less safe. The Joint CaliforniaPolicy Working Group on AI Frontier Models warns against this level of disclosure. General details about risks of foundation models can be made public withoutundermining security, especially if these risks have been demonstrated in otherfoundation models or AI technologies. Specific details about concrete vulnerabilitiesshould be disclosed carefully, with advanced notice to actors in the supply chain whoare able to remediate them prior to broader disclosure.³¹ Requiring developers to justify redactions is less effective than not requiring developers todisclose any information that would include trade secrets, cybersecurity information, or otherconfidential or proprietary information. SB 53 unnecessarily re-writes California Whistleblower law for just one industry As amended, SB 53 rewrites California’s already robust whistleblower protections for justone industry. Creating a special, one-off standard for a single sector not only sets a poorprecedent but also risks confusion and inconsistency across industries. Current law coverswhistleblowing activities associated with AI safety because there is a robust body of existinglaw that governs whistleblower protection covering employees who report violations ofstate/federal laws, rules, or regulations. These laws are intentionally tied to actions that areillegal so there are clear lines of what is considered applicable and understood who gets protection when reporting. These protections cover activities associated with AI withoutcreating unnecessary and confusing new processes in state law.> For example, Labor Code Section 6310 already protects whistleblowers who report unsafeworking conditions or work practices. Similarly, federal laws such as the Sarbanes-Oxley Actprotect employees who report safety violations or substantial and specific dangers to publichealth or safety. A brightline threshold is needed for what activity is covered so it is clearwhen a developer’s activities should be reported. For instance, in the field of research anddevelopment, innovations are being experimented with in novel contexts where there may besignificant disagreement on what actions constitute risk. Thus, the bill mandates that there bean allegation of “specific and substantial danger to public health or safety resulting from acatastrophic risk,” the inherently subjective nature of these terms leaves room for differinginterpretations as to what does or does not meet the threshold.

SB 53 requires steep penalties that are disproportionate for technical errors, inflexibleincident reporting requirements, and no right to cure

As amended, SB 53 imposes a $1 million fine for a possible paperwork error which isexcessive and risks punishing good-faith developers for technical mistakes rather thandeterring real harm. Penalties should be fair, targeted, and proportionate. As we pointed outin our July 12th letter, SB 53 requires incident reporting within 15 days but does not provideflexibility for an investigation timeline. Even if 15 days is a reasonable reporting period, requirements should be flexible because all facts may not be known within 15 days ofdiscovery. With respect to enforcement, we again state our view that the bill should grantbusinesses at least a 60 day right to cure, to ensure that law focuses on compliance and notpunishment. In addition, given the highly detailed requirements of the bill as drafted, wethink enforcement efforts should be focused on material failures to comply rather than alsocovering technical paperwork errors. While we understand your focus on this issue and appreciate the recent amendments havemade meaningful improvements to the prior version of the bill, given the immense promiseof this technology, we believe that the bill would benefit from a focus on a risk-basedframework for all frontier models, additional clarity in responsibilities among actors in the AIvalue chain, additional safeguards for trade secrets and security, and reasonable timelines, penalties, and enforcement provisions. [ …]

REGISTERED SUPPORT / OPPOSITION:

Support

Ai for Animals
Ai Futures Project
Ai Lab Watch
Ai Policy Tracker
All Girls Allowed
Anthropic Pbc
Apart Research
Association for Long Term Existence and Resilience (ALTER)
Berkeley Existential Risk Initiative (BERI)
California Federation of Labor Unions, Afl-cio
Center for Ai and Digital Policy
Center for Ai Policy
Center for Digital Democracy
Center for Human-compatible Ai
Center for Youth and Ai
Children’s Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
Design It for US
District Council of Iron Workers of the State of California and Vicinity
Earningsstream LLC
Economic Security California Action
Elicit
Encode
Encode Ai Corporation
Encode Justice
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Future of Life Institute
Indivisible California Statestrong
InnovateEDU
Momentum
Mothers Against Media Addiction
Nonlinear
Noso November
Oakland Privacy
Parents Television and Media Council
Parents Together Action
Public Interest Privacy Center
Redwood Research
Rights4girls
Scorecard
Secure Ai Future
Secure Ai Project
SEIU Califonia
Tech Oversight California
Techequity Action
The Brandes Lab At Nyu
The Midas Project
The Signals Network
Transparency Coalition.ai
Trevi Digital Assets Fund
University of California
Young People’s Alliance
Youth Power Project

Opposition

Business Software Association
Chamber of Progress
Consumer Technology Association
Los Angeles County Business Federation (BIZFED) (UNREG)
Silicon Valley Leadership Group

Oppose Unless Amended

California Chamber of Commerce
Computer and Communications Industry Association
Insights Association
Technet

Analysis Prepared by: John Bennett / P. & C.P. / (916) 319-2200

Footnotes

For a discussion of issues with defining frontier models, see “California Report on Frontier AI Policy” (June 17, 2025), pp. 36-40, https://www.cafrontieraigov.org/. ↩
Pablo Arredondo, “GPT-4 Passes the Bar Exam: What That Means for Artificial Intelligence Tools in the Legal Profession” (Apr. 19, 2023), https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/. ↩
Id. at p. 44. ↩
Introducing OpenAI o3 and o4-mini, OpenAI (Apr. 16, 2025), https://openai.com/index/introducing-o3-and-o4-mini/. ↩
Sam Altman, The Intelligence Age (Sept. 23, 2024), https://ia.samaltman.com/. ↩
Kyungtae Kim, “What is AGI, and when will it arrive?: Big Tech CEO Predictions” (Mar. 20, 2025), https://www.giz.ai/what-is-agi-and-when-will-it-arrive/; see also Kokotajlo et al, “AI 2027,” (Apr. 3, 2025), https://ai-2027.com/. ↩
John Markoff, “The Coming Superbrain,” New York Times (May 23, 2009), www.nytimes.com/2009/05/24/weekinreview/24markoff.html. ↩
International AI Safety Report, supra, at pp. 17-21. The report does not address Lethal Autonomous Weapon Systems, which are typically narrow AI systems specifically developed for that purpose. (See id. at pp. 26-27.) ↩
Cat Zakrzewski and Pranshu Verma, “New Hampshire opens criminal probe into AI calls impersonating Biden,” Washington Post, February 6, 2024, www.washingtonpost.com/technology/2024/02/06/nh-robocalls-ai-biden/. ↩
F. Salvi, M. H. Ribeiro, R. Gallotti, R. West, “On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial,” arXiv [cs.CY] (2024); http://arxiv.org/abs/2403.14380. ↩
Angela Yang, “Researchers secretly infiltrated a popular Reddit forum with AI bots, causing outrage,” NBC News (Apr. 29, 2025), https://www.nbcnews.com/tech/tech-news/reddiit-researchers-ai-bots-rcna203597. ↩
Dan Hendrycks, Introduction to AI Safety, Ethics, and Society, p. 11, https://drive.google.com/file/d/1uph559WASR4MEn6M_7Mb3lqQTapC_gZ/view?pli=1. ↩
International AI Safety Report, supra, at p. 72. ↩
Soice et al, “Can large language models democratize access to dual-use biotechnology?” https://arxiv.org/pdf/2306.03809. ↩
Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery.” In: Nature Machine Intelligence 4 (2022), pp. 189–191. ↩
Brian Christian, The Alignment Problem: Machine Learning and Human Values (Norton 2020, 1st ed.), pp. 9-11. ↩
International AI Safety Report, supra, at pp. 100-107. ↩
OpenAI, “GPT-4 System Card,” https://cdn.openai.com/papers/gpt-4-system-card.pdf. ↩
Meinke et al, “Frontier Models are Capable of In-Context Scheming,” arXiv (Jan. 2025), https://arxiv.org/pdf/2412.04984. ↩
Harry Booth, “When AI Thinks It Will Lose, It Sometimes Cheats, Study Finds,” Time (Feb. 19, 2025), https://time.com/7259395/ai-chess-cheating-palisade-research/. ↩
System Card: Claude Opus 4 & Claude Sonnet 4, pp. 27, https://www.anthropic.com/claude-4-system-card. ↩
Id. at pp. 123-126. ↩
Bommasani and Singer et al. “The California Report on Frontier AI Policy.” The Joint California Policy Working Group on AI Frontier Models. June 17, 2025. p. 39. ↩
Id. at p. 26. ↩
Id. at p. 19. ↩
Id. at p. 21. ↩
Id. at p. 35. ↩
Id. at p. 29. ↩
RAISE Act can be found at https://www.nysenate.gov/legislation/bills/2025/A6453/amendment/A. ↩
Final Report at p. [page number not specified in source]. ↩
Id. at p. 30. ↩

Assembly Privacy Committee Analysis of SB 53 (September 5, 2025)

SYNOPSIS

EXISTING LAW:>

COMMENTS:>

REGISTERED SUPPORT / OPPOSITION:

Support

Opposition

Oppose Unless Amended

Footnotes

Footnotes