MIT AI Risk Repository

AI Agents / Agentic AI

Autonomous or semi-autonomous systems that use AI to perceive, reason, and act toward goals—often coordinating across tools, APIs, or environments to perform complex, multi-step tasks on behalf of users.

AI Anomaly Detection

The process of identifying unusual or unexpected behavior in AI systems, inputs, or outputs—often signaling potential security threats, performance issues, or compliance violations.

AI Bias

Systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one arbitrary group of users over others.

AI Data Governance

AI data governance involves the management of data quality, legality, and ethical standards to ensure responsible AI deployment.

AI Detection & Response

A security practice focused on monitoring, detecting, and responding to threats targeting AI systems—including misuse, attacks, and policy violations—during training, deployment, and live operation.

AI Explainability

The degree to which humans can understand how an AI system makes decisions. Important for trust, transparency, debugging, and regulatory compliance—especially in sensitive or high-stakes applications.

AI Governance

A structured approach to overseeing the security, compliance, and responsible use of AI systems—ensuring that AI operations align with organizational policies, regulations, and ethical standards.

AI Guardrails

Controls and constraints built into AI systems to prevent harmful, unethical, or unintended outcomes. Guardrails enforce safe behavior during training, deployment, and real-time operation.

AI Hallucination

A phenomenon where AI systems, especially language models, generate confident but false, misleading, or fabricated information not grounded in reality or available data.

AI Impact Assessment

AI Impact Assessment is a critical process that evaluates the potential effects and implications of AI technologies on society, the environment, and the economy.

AI Input Manipulation

A tactic where attackers or users intentionally craft inputs to alter or exploit an AI model’s behavior. Often used to bypass filters, trigger harmful outputs, or expose vulnerabilities.

AI Inventory

Comprehensive cataloging and management of all artificial intelligence systems, models, datasets, and related resources within an organization.

AI Jailbreak

A technique used to bypass restrictions or safety filters in AI systems—causing them to generate disallowed, dangerous, or otherwise restricted content.

AI Red Teaming

Security practice where experts simulate attacks on AI systems to identify vulnerabilities, assess their robustness, and improve their resilience against real-world threats.

AI Resource Exhaustion Attacks

A denial-of-service tactic where attackers overload an AI system’s compute, memory, or token limits—causing slowdowns, failures, or forced shutdowns of models or APIs.

AI Runtime Protection

A security layer that monitors and defends AI systems while they operate—detecting threats, enforcing policies, and responding to misuse or drift in real time.

AI Security Operations (AISecOps)

A security operations discipline focused on protecting AI systems from attacks, misuse, drift, and exposure—combining real-time monitoring, threat detection, and policy enforcement across the AI lifecycle.

AI Security Posture Management

Continuously monitoring, assessing, and improving the security measures and defenses of AI systems to ensure they are protected against threats and vulnerabilities.

AI Supply Chain Security

Safeguarding the integrity and security of AI systems and their components throughout the supply chain, from development to deployment, to prevent vulnerabilities and ensure trustworthiness.

AI TRiSM

A framework that combines Trust, Risk, and Security Management to govern AI systems—ensuring they operate responsibly, ethically, and within regulatory and organizational boundaries.

AI Toxicity

The generation of harmful, offensive, or biased content by AI systems—often due to flawed training data, poor filtering, or misaligned model objectives.

AI Training Data

The datasets used to teach AI models how to recognize patterns, make decisions, or generate content. Training data quality and composition directly affect model performance, bias, and security.

API Security

Protecting Application Programming Interfaces (APIs) from vulnerabilities, ensuring secure data exchange, and preventing unauthorized access or malicious activities.

Adversarial Machine Learning

A tactic where attackers craft inputs designed to fool AI models into making incorrect predictions. These subtle manipulations pose serious risks to model reliability, accuracy, and trust.

Agentic App Security

The protection of autonomous AI systems—such as agents or AI-driven workflows—that independently perform tasks, access tools, or make decisions on behalf of users.

Application Security Posture Management (ASPM)

Tools and processes for continuously monitoring, assessing, and improving the security posture of software applications throughout their development lifecycle, with a focus on identifying, assessing, and mitigating vulnerabilities and risks associated with applications to ensure they remain secure against potential cyber threats.

Application Security Testing

Evaluating applications to identify and mitigate security vulnerabilities, ensuring that they are secure against threats and comply with security standards.

Artifact Repository

A storage location for software artifacts, including binaries and dependencies, used in the development and deployment process.

Artificial Intelligence (AI)

Technology that enables machines to perform tasks typically requiring human intelligence, such as reasoning, decision-making, and learning. Used in automation, analytics, and interaction across diverse business and consumer applications.

Audit Trail

A chronological record of all activities and changes that have occurred in a system or process, ensuring transparency and accountability.

Automated Security Testing

The use of automated tools and processes to continuously test applications for security vulnerabilities throughout the development lifecycle.

Bug Bounty

A a program that rewards individuals for finding and reporting security vulnerabilities in an organization's software, encouraging proactive identification and resolution of potential security issues.

CISA Known Exploited Vulnerabilities (KEV)

List compiled by the US Cybersecurity and Infrastructure Security Agency which identifies vulnerabilities confirmed as being exploited in the wild.

California AI Act

A proposed California law that aims to regulate AI systems—focusing on safety, transparency, and civil rights protections—especially in high-risk and public-facing applications.

Code Repository

A storage location for source code, often using version control systems like Git.

Common Vulnerabilities and Exposures (CVE)

A list of publicly know vulnerabilities maintained by NIST.

Common Vulnerability Scoring System (CVSS)

A standardized, repeatable, and vendor agnostics method to compare application vulnerabilities.

Common Weakness Enumeration (CWE)

A community-developed list of common software and hardware weaknesses, which could introduce vulnerabilities.

Compliance

Adhering to industry standards, regulations, and best practices to ensure that applications meet legal and security requirements

Compliance Audit

A comprehensive review of an organization's adherence to regulatory guidelines.

Compliance Risk

The potential for losses or legal penalties due to violations of laws, regulations, or prescribed practices.

Container

A lightweight, standalone executable package of software that includes everything needed to run it, ensuring consistency across environments.

Contextual Risk Engine (CoRE)

AppSOC�s proprietary technology for prioritizing security issues based on severity, exploitability, and business context.

Continuous Integration/Continuous Deployment (CI/CD)

A practice that automates the integration and deployment of code changes, enabling frequent and reliable updates to applications.

Continuous Threat Exposure Management (CTEM)

A proactive and continuous program to monitor, evaluate, and reduce levels of exploitability and validate analysis and remediation processes.

Data Poisoning

A training-time attack where malicious or manipulated data is introduced into the training set to distort a model’s behavior, reduce accuracy, or embed hidden backdoors.

Data Privacy

The aspect of information technology that deals with the ability of an organization or individual to determine what data can be shared with third parties.

Dependency Resolution

The process of determining and fetching the correct versions of dependencies for a software project.

DevOps

A methodology that combines software development (Dev) and IT operations (Ops) to shorten the development lifecycle, deliver high-quality software continuously, and improve collaboration between development and operations teams.

DevSecOps

An approach that integrates security practices into the DevOps process, ensuring that security is incorporated at every stage of the software development lifecycle to enhance the overall security posture of applications.

Dynamic Application Security Testing (DAST)

A testing method that analyzes applications in their running state to identify vulnerabilities by simulating external attacks.

EU AI Act

A comprehensive European regulation that classifies AI systems by risk and imposes strict requirements on high-risk applications—covering transparency, accountability, and safety.

Endpoint Detection and Response (EDR)

A system that monitors and analyzes endpoint activities to detect, investigate, and respond to security incidents in real-time.

Exploit Prediction Scoring System (EPSS)

A process developed by First.org for estimating the likelihood that a software vulnerability will be exploited in the wild.

False Negative

A failure to detect an actual vulnerability or threat, leading to a potential security risk remaining unaddressed.

False Positive

A security alert that incorrectly indicates the presence of a vulnerability or threat when none exists.

General Data Protection Regulation (GDPR)

European Union regulation that mandates strict data privacy and security measures for protecting personal data, giving individuals greater control over their personal information.

Generative AI (GenAI)

AI systems designed to create original content—such as text, images, code, or audio—by learning patterns from large datasets. Used in automation, creativity, and user interaction across industries.

Generative Artificial Intelligence (Gen AI)

Artificial intelligence systems that create new content, such as text, images, or music, by learning patterns from existing data, exemplified by models like OpenAI's GPT-4.

Health Insurance Portability and Accountability Act (HIPAA)

U.S. law that sets national standards for the protection of sensitive patient health information, ensuring that such information is kept confidential and secure, particularly in electronic form.

Hugging Face

An open-source platform and model hub widely used for sharing, deploying, and fine-tuning machine learning and NLP models, especially transformers and large language models.

Improper Output Handling

A failure to properly inspect, filter, or validate AI-generated outputs—leading to the release of unsafe, biased, or non-compliant content into user-facing applications.

Incident Management

The process used to manage the lifecycle of incidents to ensure that normal service operation is restored as quickly as possible.

Incident Response

The process of identifying, managing, and mitigating security incidents to minimize their impact on the organization.

Infrastructure as Code (IaC)

The ability to provision and support computing infrastructure using code instead of manual processes and settings.

Integrity Check

Verification that software has not been altered or tampered with.

Internal Audit

An independent, objective assurance and consulting activity designed to add value and improve an organization's operations.

Key Risk Indicators (KRIs)

Metrics used to provide an early signal of increasing risk exposures in various areas of an enterprise.

LLMOps

A set of practices and tools for managing the lifecycle of large language models (LLMs), including deployment, monitoring, scaling, and governance in production environments.

Large Language Model (LLM)

A type of AI model trained on massive text datasets to understand and generate human-like language. LLMs power chatbots, assistants, and other natural language applications.

Large Language Models (LLM)

Advanced AI systems, such as OpenAI's GPT-4, that generate human-like text by processing and understanding vast datasets, enabling applications from automated customer service to content creation.

License Compliance

Ensuring that software components comply with licensing agreements and open-source licenses.

MITRE ATLAS

A knowledge base that catalogs the tactics, techniques, and case studies of adversarial attacks on machine learning (ML) and artificial intelligence (AI) systems to help organizations understand and mitigate these threats.

MITRE ATT&CK

Knowledge base of adversary tactics and techniques based on real-world observations, used as a foundation for developing threat models and methodologies in the cybersecurity community.

MLOps

The practice of deploying, managing, and monitoring machine learning models in production to ensure they operate efficiently and effectively.

Machine Learning (ML)

A method that enables software systems to learn from data and improve performance without explicit programming. Commonly used for predictions, classifications, and decision-making across dynamic or data-rich environments.

Membership Inference Attack

An attack that attempts to determine whether specific data was part of an AI model’s training set—potentially exposing sensitive or private information about individuals or proprietary records.

Model Context Protocol (MCP)

A structured format for managing inputs, context, and control signals passed to AI models—enabling more secure, transparent, and auditable language model interactions.

Model Deployment

The process of integrating a machine learning model into a production environment where it can make predictions on new data and deliver business value.

Model Drift

A condition where an AI model’s performance degrades over time due to changes in data, user behavior, or environment—resulting in reduced accuracy, reliability, or alignment with business goals.

Model Extraction

An attack where adversaries attempt to replicate or reverse-engineer a deployed model by querying it—stealing intellectual property or exposing sensitive model logic.

Model Fuzzing

Testing technique used to identify vulnerabilities and weaknesses in machine learning models by inputting random, unexpected, or malformed data to observe how the model responds.

Model Inference

The process of using a trained AI model to generate predictions, classifications, or responses based on new input data. Inference occurs during model deployment and often in real-time production environments.

Model Inversion

An attack where adversaries reconstruct sensitive training data by analyzing a model’s outputs—potentially exposing private or proprietary information used during model training.

Model Monitoring

Continuously tracking the performance, accuracy, and behavior of deployed machine learning models to ensure they operate correctly and efficiently over time.

Model Scanning

The process of analyzing machine learning models for vulnerabilities, biases, and compliance with security and ethical standards to ensure they are safe and reliable for deployment.

Model Serving

The process of deploying machine learning models into production environments where they can process real-time data and generate predictions or insights.

Model Supply Chain Security

Protecting the integrity and security of machine learning models throughout their development, deployment, and operational lifecycle to prevent tampering, unauthorized access, and vulnerabilities.

NIST AI Risk Management Framework (RMF)

A U.S. government framework developed by NIST to guide organizations in managing risks associated with AI systems—emphasizing trustworthiness, accountability, and governance across the AI lifecycle.

National Institute of Standards and Technology (NIST)

An agency of the US Department of Commerce whose mission is to promote American innovation and industrial competitiveness

National Vulnerability Database (NVD)

U.S. government repository of standards-based vulnerability management data represented using the Security Content Automation Protocol (SCAP).

OWASP Top 10

A standards list from the non-profit Open Worldwide Application Security Project representing a broad consensus about the most critical security risks to web applications.

OWASP Top 10 for LLM Applications (2025)

List of the most critical security risks associated with large language models (LLMs), providing guidance for identifying, understanding, and mitigating these vulnerabilities to enhance the security and integrity of AI systems.

Open Source Software (OSS)

Software that is released with a license allowing anyone to view, modify, and distribute the source code.

Package Manager

A tool that automates the process of installing, upgrading, configuring, and removing software packages.

Patch Management

The process of distributing and applying updates to software to fix vulnerabilities and improve functionality.

Penetration Testing

A simulated cyberattack against an application to identify security weaknesses that could be exploited by malicious actors.

Personally Identifiable Information (PII)

Information that can be used on its own or with other information to identify, contact, or locate a single person, or to identify an individual in context.

Prompt Injection

A type of attack where malicious input is crafted to manipulate or alter the behavior of an AI system, particularly those using natural language processing (NLP) models.