Response to OSTP “National Priorities for Artificial Intelligence Request for Information”

The Center for a New American Security (CNAS) welcomes the opportunity to provide comments in response to OSTP’s “National Priorities for Artificial Intelligence Request for Information.” CNAS is an independent, bipartisan organization dedicated to developing bold, pragmatic, and principled national security solutions. The CNAS AI Safety & Stability project is a multi-year, multi-program effort that addresses the established and emerging risks associated with artificial intelligence.

Authors:
Timothy Fist, Fellow, Technology and National Security Program (tfist@cnas.org)
Michael Depp, Research Associate, AI Safety and Stability Project (mdepp@cnas.org)
Caleb Withers, Research Assistant, Technology and National Security Program (cwithers@cnas.org)

This document reflects the personal views of the authors alone. As a research and policy institution committed to the highest standards of organizational, intellectual, and personal integrity, CNAS maintains strict intellectual independence and sole editorial direction and control over its ideas, projects, publications, events, and other research activities. CNAS does not take institutional positions on policy issues and the content of CNAS publications reflects the views of their authors alone. In keeping with its mission and values, CNAS does not engage in lobbying activity and complies fully with all applicable federal, state, and local laws. CNAS will not engage in any representational activities or advocacy on behalf of any entities or interests and, to the extent that the Center accepts funding from non-U.S. sources, its activities will be limited to bona fide scholastic, academic, and research-related activities, consistent with applicable federal law. The Center publicly acknowledges on its website annually all donors who contribute.

1. Introduction

This document outlines a set of policy measures for harnessing the benefits and mitigating the risks of AI. We focus on questions within the category “Protecting rights, safety, and national security”. We focus on “frontier” AI models—general-purpose “foundation” models at the frontier of research and development (R&D)—for several reasons.¹ First, these models are likely to represent the forefront of AI capabilities and risks across many domains. Second, the development and deployment of these models is currently largely unregulated. Lastly, these models present a special set of challenges for policymakers that calls for dedicated guardrails on top of existing sector-specific regulation: they develop new capabilities in an unpredictable way, are hard to make reliably safe, and are likely to proliferate rapidly due to their multitude of possible uses. While the risks and challenges posed by frontier AI will often be novel, there are lessons to be learned from other sectors. For example, risk management for frontier AI will likely have a similar character to risk management in the cyber domain, which generally involves:

• A ‘security mindset’, expecting to confront unintended vulnerabilities and adversaries;
• Establishing organization-wide governance, risk management, and standards;
• Incident response capabilities;
• Integrating safeguards into the design process rather than implementing them as an afterthought;
• Promoting information sharing and collaboration to learn lessons and improve safety practices.

Our recommendations draw on the emerging literature of best practices and predicted risks for developing and deploying frontier AI models.

2. Summary of recommendations

Question 1: What specific measures—such as standards, regulations, investments, and improved trust and safety practices—are needed to ensure that AI systems are designed, developed, and deployed in a manner that protects people’s rights and safety? Which specific entities should develop and implement these measures?

As a foundation for safe and trustworthy AI, government bodies such as the National Science Foundation should provide active support and substantial funding for research into relevant technical solutions and evaluations²—in particular, research specifically applicable to frontier systems and focused on ensuring their trustworthiness with high levels of confidence (as in the National Science Foundation's recent Safe Learning-Enabled Systems solicitation). (see section 4.1)

Given the difficulty of preventing dangerous models from proliferating once they have been developed and/or released, the federal government should establish appropriate thresholds for “frontier models” and work with data center operators to ensure a transparent chain of frontier model provenance. This would involve tracking data centers capable of efficiently producing frontier models, requiring these data centers to implement a minimum set of cybersecurity standards, and requiring ‘Know Your Customer’-style diligence on users who are accessing large amounts of compute. (see section 4.2)

Question 3: Are there forms of voluntary or mandatory oversight of AI systems that would help mitigate risk? Can inspiration be drawn from analogous or instructive models of risk management in other sectors, such as laws and policies that promote oversight through registration, incentives, certification, or licensing?

Frontier AI labs should adopt the following best practices to mitigate risks throughout the frontier AI development and deployment life cycle:

• Pre-training and pre-deployment risk assessments and model evaluations; (see sections 4.3 and 4.4)
• Accounting for the risk profiles of different deployment strategies (e.g. staged release and structured access); (see section 4.4.2)
• Strong, independent internal audit functions; (see section 4.5.2)
• Strong cybersecurity practices, with regard to relevant standards. (see section 4.5.3)

Many of these practices can, and have been, voluntarily adopted by frontier labs. The federal government should encourage their adoption by requiring AI and cloud computing companies to adopt these best practices for frontier models as a condition of government contracts. (see section 5.4)

Congress should establish a regulatory body specifically focused on frontier AI. This body should have the authority to enforce minimum standards for the development and deployment of frontier AI systems. (see sections 5.1 and 5.2)

Congress should also consider whether current liability laws adequately account for the challenges of frontier AI systems—such as opaque internal logic, growing autonomy, the wide diffusion of powerful capabilities, and a yet-to-emerge consensus on what constitutes reasonable care in their development and deployment. (see section 5.3)

Question 7: What are the national security risks associated with AI? What can be done to mitigate these risks?

The government should focus attention on safety measures for highly capable, general-purpose foundation models and develop technical thresholds to categorize such systems based on whether they could pose meaningful risks to public safety. Three key areas where such systems are likely to pose national security risks in the future are dual-use science, cyber operations, and deception/persuasion operations. (see section 3.1)

3. Defining frontier AI and its risks

We use the term “Frontier AI” to refer to foundation models at the frontier of capabilities that are powerful enough to have a meaningful chance of causing severe harm to public safety and national security.³ Recent years have seen remarkable improvements in the capabilities of foundation models, following empirically derived ‘scaling laws’⁴ which show that models steadily improve as they are made larger and trained using more computation. However, specific qualitative capabilities can emerge suddenly or improve dramatically as systems are scaled up,⁵ making it difficult to predict the exact nature and timeline of future benefits and dangers from AI systems. This necessitates proactive policymaking to prepare for potential capability enhancements in the most critical domains. We highlight three domains below where results from today’s most powerful AI models imply that future generations of these models could pose serious risks. However, the dual-use nature of such models makes for a thorny definitional challenge: how can regulators demarcate frontier models from those that clearly do not pose meaningful risks to public safety and national security?

At present, there does not exist a method for reliably establishing whether a planned model (based on its architecture, training data, training task, computational inputs, etc.) will have a given set of dangerous capabilities. One straightforward approach might be to use computational inputs (measured in floating point operations, or FLOP), which has historically been a decent proxy for the breadth and depth of model capabilities.⁶ This approach has the advantage of being determinable ahead of time based on a model’s planned inputs. Establishing such a threshold would also allow for the identification of data centers capable of efficiently training frontier models (i.e. data centers capable of producing the threshold amount of FLOP within a certain period of time or within a certain budget), which would provide policymakers with visibility of where high-risk models are likely to be produced, and establish more transparency over their provenance. This approach also has a key downside: research progress in efficient algorithms for training new AI models will decrease the computational inputs required to build models with dangerous capabilities as time passes.

This highlights three priorities for the federal government. First, to develop a near-term set of technical thresholds for categorizing highly-capable models that are liable to possess dangerous capabilities, in consultation with technical AI experts developing these models.⁷ Second, to keep these thresholds flexible based on progress in AI R&D and the development of risk mitigations. Lastly, given the inevitable downsides of any threshold based on a technical proxy for capabilities, build a system of regulatory guardrails for developing and deploying powerful AI systems that use model-specific risk assessments and evaluations.

3.1. Emerging Risks from AI

Current AI systems demonstrate emerging capabilities that could have destabilizing effects on international relations⁸ and lower the barrier to entry for non-state actors to cause harm.⁹

3.1.1. Weaponizable scientific research and manufacturing

Certain dual-use scientific research directions present both benefits and national security risks—for example, the development of chemical and biological compounds. The U.S. government accounts for these risks in policies and regulations, such as in Life Sciences Dual Use Research of Concern or ITAR restrictions on toxicological agents (along with algorithms and models supporting their design or deployment). However, these policies generally limit their focus to tools specifically designed for these domains, which could prove problematic as generative AI models increasingly demonstrate relevant capabilities. To date, key examples of how such tools could be used to cause harm include the following:¹⁰

• In March 2022, researchers took MegaSyn, a generative AI tool for discovering therapeutic molecules, and ‘inverted’ it, tasking it with finding molecules that harm the nervous system.¹¹ It found tens of thousands of candidate molecules, including known chemical weapons and novel molecules predicted to be as or more deadly.
• In April 2023, researchers gave a GPT-4 powered system access to the internet, code execution, hardware documentation, and remote control of an automated ‘cloud’ laboratory. It was “capable of autonomously designing, planning, and executing complex scientific experiments”, and in some cases, was willing to outline and execute viable methods for synthesizing illegal drugs and chemical weapons.¹²

3.1.2. Cyber operations

Most software has dormant security vulnerabilities; finding them for offensive or defensive purposes¹³ is a matter of resourcing, time, effort, and skill. AI can automate relevant activities:¹⁴ detecting and responding to threats, finding and fixing vulnerabilities, and launching offensive attacks. These capabilities can support cyber defense and national security when used by the U.S. government and other responsible actors, but they can also cause significant harm when used by malicious actors. Large foundation models are already highly capable at coding and excel in data-rich domains—attributes which support their effectiveness in cyberspace. They can also be deployed at scale to target many systems at once and can be flexible, re-writing code on the fly to avoid detection and look for sensitive data. So far, there are several public examples of how these capabilities can be used:

• ‘Polymorphic’ malware is a type of malicious software that can change its code to avoid being detected. BlackMamba is a polymorphic keystroke logger that avoids detection by not having any hardcoded keylogging capabilities.¹⁵ Instead of having a fixed method of stealing information from users’ keyboards, it requests new keylogging code from ChatGPT every time it runs. This makes it harder to be stopped by security programs that look for specific segments or patterns of malicious code.
• DarwinGPT is a proof-of-concept virus that uses ChatGPT to modify its code and act autonomously. It has the goal of spreading itself to other systems. It can create new functions to help it achieve this goal without any specific instructions.
• Similarly, PentestGPT uses ChatGPT to automate penetration testing—the process of testing a system’s security by simulating an attack.¹⁶
• ChatGPT can be used to generate plausible ‘‘spear phishing” emails, which target specific individuals with personalized messages that lure them to click on malicious links.¹⁷
• GPT-4 can help identify vulnerabilities in computer code.¹⁸

3.1.3. Deception/persuasion operations

Information operations, propaganda, and manipulation—and defense against them—have long played a critical role in international relations and national security, as state and non-state actors look to influence perceptions, behaviors, and decision-making processes at various levels. In the contemporary digital landscape, these operations can now be conducted at unprecedented scales and speeds. With AI systems increasingly able to generate human-like text and realistic media, this trend will be accelerated, as adversaries look to employ AI’s potential for increasingly bespoke and adaptable operations while retaining scale. Generative AI has demonstrated capabilities for deception and persuasion:

• Meta’s Cicero AI achieved human-level performance in Diplomacy, a strategy game centered around natural language negotiation and alliance building.¹⁹
• An initial version of GPT-4 was tasked with getting a human worker on TaskRabbit to solve a CAPTCHA (an online test used to distinguish between humans and automated systems) for it. When challenged on whether it was a robot, it recognized it needed to come up with an excuse and pretended to be a visually impaired human.²⁰
• GPT-3 performed at human levels in crafting persuasive messages on contentious political issues.²¹

Widely-available generative AI can also be used to fabricate multimedia such as deepfake images²² or speech mimicking someone’s voice²³. The most capable models have advantages in generating compelling and personalized content:

• For image generation, the largest and most capable models perform better at incorporating “world knowledge, specific perspectives, or writing and symbol rendering”.²⁴
• Larger models are better able to model and tailor their answers to users’ preferences. Larger language models, especially when ‘fine-tuned’ to provide answers that humans prefer, tend to be more ‘sycophantic’, forgoing moderation and consistency to give answers that correspond to the user’s apparent political views.²⁵

4. Protecting rights, safety, and national security across the AI lifecycle

The U.S. government must work with industry to establish safeguards across the entire frontier AI lifecycle, from gathering data through to releasing a trained model. Safeguards are needed well before frontier models are released: if a trained model is released for download, it becomes highly difficult to track and regulate. A trained model is an easy-to-copy piece of software, with capable models presenting an attractive target for theft. Additionally, once someone has access to a downloaded model, it is possible to ‘fine-tune’ the model to elicit dangerous new capabilities with minimal additional computation relative to what was required for initial training. This poses a problem for oversight, as the fine-tuning process can involve the elicitation of dangerous capabilities that were previously not present—for example, if a conversational AI were fine-tuned to specialize in generating cyber-offensive code.

4.1. Laying the foundations: technical research into safeguards

As frontier AI models continue to rapidly advance, the engineering of reliable safeguards remains an open challenge:²⁶ frontier models can currently be coaxed into providing dangerous information²⁷, have largely inscrutable internal reasoning,²⁸ and are prone to “hallucination” (making up facts)

Desirable qualities of extreme risk evaluations, adapted from Shevlane et al., “Model evaluation for extreme risks,” arXiv (2023), https://arxiv.org/abs/2305.15324;

Model evaluation should involve scientific and policy experts in domains such as biosecurity and cyber operations, to help contextualize the risk posed to society by given capabilities. To this end, federal agencies with relevant expertise should consider collaborating with frontier AI evaluators and standards bodies—for example, CISA and/or NSA could offer guidance on the types of cyber capabilities that could pose serious risks to national security if available in released models.

Mitigating evaluation gaming

In implementing and interpreting model evaluations, risks from evaluations being gamed should be accounted for (this could be intentional or inadvertent on the part of the developer).⁴⁰ If labs optimize for demonstrating safety through particular evaluations, these evaluations may lose validity or usefulness as indicators. Measures to reduce this risk can include:

• Avoiding directly training models to pass evaluations;
  • Using canary strings: some benchmarking and evaluation projects include unique identifying text called ‘canary strings’ in associated documents—these identifiers help developers exclude these documents from training datasets (and help evaluators detect if they have been included);
• Keeping some details of some evaluations non-public;
• Research into methods for identifying if a model is ‘gaming’ evaluations and tests.⁴¹

Involving external evaluators

Given AI labs’ conflict of interest in evaluating their own models, third parties should also be involved. Evidence from other domains suggests that third-party evaluators should:⁴²

• have well-scoped objectives;
• be professionalized—for example through training, standardized methodologies, and third-party accreditation;
• have a high degree of access to models—direct access to model outputs at a minimum, and potentially information about training data and model architecture, and the ability to fine-tune models⁴³;
• maintain independence—for example, avoiding compensation schemes that are tied to evaluation results, or cross-selling of other services.

Involving a more diverse range of evaluators can also help to identify risks more comprehensively.⁴⁴ Frontier AI labs can demonstrate responsibility by taking these factors into account as they determine the involvement of external evaluators. Since the frontier model evaluation ecosystem is still emerging, the federal government should consider supporting the development of relevant standards and an accreditation body.

4.4. Safeguards for model deployments

4.4.1. Pre-deployment risk assessment

Frontier AI developers should undertake risk assessments before deployment. In addition to supporting internal decision-making, these could be shared with external evaluators and researchers, regulators, or publicly. Risk assessments should include evaluation results, alongside the developer’s justification for why the deployment is safe given those results.”⁴⁵ At a minimum, these assessments should also reflect the NIST AI Risk Management Framework and associated resources.⁴⁶ Pre-deployment risk assessments should also account for the capabilities of systems that are already publicly available (including the cost and ease of employing these capabilities): if systems are already widely available with a particular capability/risk profile, the risk from additional systems with this same profile could be marginal.

4.4.2. Deployment strategy

Frontier AI labs should account for the risk profile of their deployment strategy. In particular, they can reduce risks by providing users with structured access⁴⁷ to models through web interfaces and APIs, instead of allowing them to download the trained AI models directly. This:

• allows labs to implement ‘Know Your Customer’ policies;
• allows labs to monitor for misuse of their models;
• allows restriction of access to the models if concerning vulnerabilities, capabilities, or synergies with external tools are discovered post-deployment;
• prevents modification of the underlying model, such as through fine-tuning, to remove safeguards or introduce new dangerous capabilities;⁴⁸
• helps protect intellectual property, such as innovations in model architecture, which could otherwise help strategic competitors of the US to ‘catch up’ in frontier AI capabilities.

Risks can also be reduced by staging the release of models: first testing them with relatively small and trusted groups of users before making them available more widely.⁴⁹ This can help reduce the chance that unanticipated vulnerabilities are exploited by bad actors or cause harm at scale.

Where frontier labs do pursue more open release strategies, this should be accounted for in their risk assessments. For instance, risks from malicious actors become greater if a model is widely released, and risks from unattributable use or fine-tuning become greater if a model is open-sourced.

4.5. Further considerations for guardrails throughout the AI lifecycle

4.5.1. Accounting for iteration in the AI lifecycle

Though this document includes references to discrete phases such as training and deployment, in practice the lifecycle of frontier AI systems will often be fluid, iterative, or overlapping. For example, the latter stages of training frontier models will often involve some external availability, like making models available to crowdworkers to enable training on human feedback. Additionally, systems may be updated frequently or in real-time based on user interaction following their deployment. For AI systems that are subject to continuous refinement, new rounds of risk assessments could be triggered following significant additional training computation and/or improvement on performance benchmarks.⁵⁰

Additionally, risk assessments may need to be revisited after models have been deployed more widely: users may uncover new capabilities or vulnerabilities, or new tools and datasets may become available to the system. Just as the FDA conducts postmarketing surveillance of medical products after they have been released on the market, frontier AI models will need similar efforts to track and mitigate risks that become apparent after deployment.

4.5.2. Internal audit functions

Frontier AI labs should have internal audit functions that are specific to the risks from these systems and accounting for the full AI lifecycle. The value of internal audit has been recognized in, for example, requirements for public companies through the Sarbanes-Oxley Act or federal agencies per the Government Accountability Office’s ‘Green Book’. Functioning with a degree of independence, internal audit functions should evaluate an organization's risk profile and report to the executive team. They should understand risk management literature and field-specific standards, especially regarding AI. The mandate for these boards should ensure they have sufficient authority, including access to models and associated documents, to conduct their evaluations effectively.⁵¹

4.5.3. Cybersecurity

Frontier labs should put significant effort into cybersecurity. As economically valuable pieces of software that can be easily shared and replicated, frontier AI models are likely to be targets for theft and hacking.

At the very least, labs developing frontier models should adhere to basic cybersecurity standards such as the NIST Cybersecurity Framework or ISO 27001. Cybersecurity should also be considered in the implementation of efforts to mitigate risks from frontier models: for example, where regulators or evaluators require access to frontier models, secure onsite access will generally be preferable to distributing models over the internet.

4.5.4. Accounting for risks from transparency

In general, transparency from frontier AI labs around their models can help support accountability and trust. However, it should be noted that:

• Full transparency around data sources, training methods, and model architecture (let alone model weights themselves) would often involve the disclosure of proprietary secrets that could help strategic competitors of the US to ‘catch up’ in frontier AI capabilities;
• Full transparency around model evaluations and capabilities could draw attention to and support the development of dangerous capabilities by bad actors and strategic competitors, or intensify security dilemma dynamics;
• Full transparency around how model evaluations are conducted could allow them to be ‘gamed’ by developers.⁵²

In some cases, only regulators or external evaluators should have full access to these details (with appropriate commitments to confidentiality), with the public instead receiving high-level conclusions or summaries.

5. Regulatory infrastructure

5.1. Potential roles for federal agencies

The regulatory challenge posed by frontier AI models would benefit from a dedicated regulatory body: to help ensure clear accountability, responsiveness to the rapid pace of technological advancements, a critical mass of technical expertise, and holistic oversight of the entire AI lifecycle for these models. Congress should establish such a body.

The most important reason for a dedicated regulatory approach is the general-purpose nature of frontier AI: any sector-specific regulatory bodies would inevitably overlap significantly with each other, given they would need to oversee the development and deployment processes of the same general-purpose models. A dedicated regulatory body should have sufficient resourcing, expertise and information access powers to oversee labs developing frontier AI systems and the associated accountability ecosystem, and have the remit to focus especially on societal-scale risks. Although this body would have overarching responsibility, it should not preclude interagency collaboration, or the possibility of assigning specific support or enforcement roles to other organizations.

In considering agency roles around regulating frontier AI models, several factors should be taken into account. These may include, but are not limited to:

• The overlap between regulating advanced AI models and the Department of Commerce’s role in controlling the exports of high-tech AI chips used for their training;
• NIST’s role in developing and promoting relevant standards;
• The Department of Energy’s experience in handling scientific advancements with national security implications, such as advanced computing;
• The support the Intelligence Community can provide in assessing whether releasing model architectures or weights would aid the AI initiatives of potential rivals, and in identifying misuse of frontier AI systems.

Enhanced regulatory focus on frontier AI models should not be at the expense of existing regulators overseeing relevant use of AI systems—frontier or otherwise. As emphasized in a recent statement by the Consumer Financial Protection Bureau, Justice Department’s Civil Rights Division, Equal Employment Opportunity Commission, and Federal Trade Commission, “Existing legal authorities apply to the use of automated systems and innovative new technologies just as they apply to other practices.”⁵³ The National AI Advisory Committee's Year 1 Report includes a number of recommendations that would support the U.S. Government’s overall support of AI accountability.

5.2. Licensing

Licensing requirements are common in high-risk industries, such as aviation⁵⁴ and drug manufacturing⁵⁵. A federal regulatory body should license the deployment of frontier AI, given the potential future risks that these systems could pose, and the fact that such systems are likely to proliferate widely in a hard-to-control fashion if released for download. Model development would also ideally be included in a licensing regime: “model deployment” is not well-defined in many contexts,⁵⁶ and models (as pieces of software) may proliferate through theft or internal leaks once developed.

The most straightforward licensing regime to create and enforce for AI development would require developers planning to use more than a particular amount of compute during a training run to first obtain a license to do so. Technical thresholds (compute-based or otherwise) for requiring a license should be set so as to only apply to systems that could pose meaningful risks to public safety, as described in section 3. The amount of compute triggering licensing requirements could change over time, slowly increasing in line with AI capabilities and safety mitigations, to ensure that it does not stifle innovation from smaller companies or those pursuing capabilities that are less dangerous.

Licenses themselves could be granted based on the developer demonstrating compliance with a particular set of safety standards, such as conducting thorough risk assessments and engaging in external evaluations (see sections 4.3 and 4.4). The establishment of these standards should be initiated and sustained by policymakers, but driven by technical experts, including AI developers, academic AI researchers, and AI safety & ethics experts. NIST’s AI Risk Management Framework is a promising starting point for these efforts.

5.3. Liability

Liability rules can help incentivize those developing and deploying AI systems to voluntarily implement appropriate risk management practices. While mandatory requirements will also be needed to adequately address severe risks and account for the difficulty of comprehensively identifying downstream harms, liability can help ensure AI labs take full advantage of their technical expertise and implement safeguards that may otherwise take time to be accounted for by regulators. Given potential challenges from frontier AI systems—opaque internal logic⁵⁷, an expanding ecosystem of autonomous decision-making by internet-connected models⁵⁸, wide availability to users of varying means to compensate any victims of AI-fueled harms, and yet-to-emerge consensus on what constitutes reasonable care—Congress should consider whether current liability laws are fit-for-purpose when applied to fronter AI models. Liability models that could help address these challenges could include vicarious liability, strict liability, joint and several liability, or industry-funded compensation funds.

5.4. Government procurement

Procurement rules are a promising avenue to encourage the adoption of best practices in frontier AI labs; the federal government should set minimum standards for frontier AI labs receiving government contracts. As a likely significant customer of these advanced AI systems, the U.S. government has a vested interest in ensuring their trustworthiness and mitigating potential risks. Because of the powerful incentive of government contracts, U.S. government requirements are likely to see broad adoption among leading frontier labs, providing a potential middle ground between voluntary and mandatory standards. While specific agencies may have their own unique requirements, the U.S. government should generally strive for standardized rules regarding frontier AI procurement across agencies, such as through Federal Acquisition Regulations.