AI Safety & Security
Ecosystem

AI security and model validation platform acquired by Cisco in 2024. Provided runtime protection, model scanning, and adversarial robustness testing. Now integrated into Cisco's security portfolio.

Creator of GPT-4 and ChatGPT. Largest frontier AI lab by revenue. Published safety frameworks but governance has weakened as commercial pressures have grown, with key safety staff departures.

Meta AI develops and releases LLaMA family of open-weight models. Largest open-source AI contributor but safety governance has been reactive, driven by external pressure rather than proactive policy.

360-degree AI security platform for enterprise GenAI adoption. Acquired by Cato Networks in September 2025. Validated the AI security acquisition thesis.

The GitHub of machine learning - hosting 500K+ models and datasets. Drives industry standards for model documentation through model cards and safety evaluations.

Dominant AI compute provider. NeMo Guardrails framework enables developers to add safety constraints to LLM applications. Controls the infrastructure backbone of AI.

IBM Research's Trustworthy AI division publishing extensively on fairness, explainability, and robustness. AI Fairness 360 and AI Explainability 360 toolkits are widely used.

Leading child safety technology organization using AI to detect CSAM and protect children online. Safer product helps platforms detect child sexual abuse material.

IBM Research's Trustworthy AI division publishing extensively on fairness, explainability, and robustness. AI Fairness 360 and AI Explainability 360 toolkits are widely used.

Major AI data infrastructure provider with Safety & Alignment Lab providing RLHF data and evaluation services. SEAL benchmarks are industry-referenced.

Enterprise data and AI platform. Acquired Mosaic ML to build foundation models (DBRX). MLflow provides model lifecycle governance and experiment tracking.

AI-powered email security detecting sophisticated social engineering and AI-generated phishing. Behavioral AI identifies threats that traditional filters miss.

Major cybersecurity platform using Charlotte AI for threat detection. Developing AI-specific security features for model protection and AI workload security.

Developer security platform expanding into AI code security. Scans AI-generated code for vulnerabilities and secures AI/ML dependencies.

Leading cloud security platform with AI Security Posture Management (AI-SPM). Detects misconfigurations, exposed models, and insecure AI pipelines. Acquired by Google for $32B.

Enterprise security leader with Precision AI for threat prevention. AI Access Security controls employee AI tool usage. AI Runtime Security protects AI workloads.

Google's combined AI research lab with one of the largest safety research teams globally. Publishes extensively on alignment, interpretability, and evaluation.

Amazon's AI safety efforts span AWS Bedrock Guardrails, responsible AI service features, and strategic investment in Anthropic for safety leadership.

Builds Claude, one of the leading AI assistants. Founded by former OpenAI researchers with a safety-first mission. Revenue from API access and enterprise subscriptions. The most commercially successful AI safety-origin company.

AI model monitoring and observability platform that tracks AI performance, detects drift, and monitors for bias in production. Like Datadog but specifically for AI models.

Enterprise AI governance platform that automates compliance documentation, risk assessment, and policy enforcement across an organisation's AI portfolio. The market leader in AI governance tooling.

Secures the entire data and AI lifecycle - from training data to production deployment. $100M Series B. Gartner AI TRiSM leader. The most comprehensive AI security platform.

Elon Musk's AI lab building the Grok family of foundation models. Focused on rapid capability development with minimal published safety governance.

Enterprise-focused LLM provider building the Command R family of models. Strong focus on RAG, multilingual capabilities, and enterprise deployment safety.

French AI lab building open-weight foundation models. Positioned as Europe's frontier AI champion but safety governance lags capability development.

Israeli AI lab building the Jamba family of foundation models for enterprise use. Combines Mamba architecture with Transformers for efficient long-context processing.

Palo Alto Networks AI Security division - incumbent cybersecurity leader extending platform into AI-specific threat detection, model security, and AI-powered SOC automation.

Enterprise AI observability platform providing model monitoring, explainability, drift detection, and fairness analytics. Stanford AI Lab roots.

Unified data intelligence platform covering privacy, security, governance, and AI compliance. AI governance module tracks data lineage in training and manages consent.

Security platform for AI copilots and low-code/no-code applications. Acquired by Microsoft. Addresses risk of ungoverned AI assistants in enterprises.

Controversial facial recognition company that scraped billions of images from the internet. Fined by multiple regulators. A cautionary case study in AI governance failure.

AI-powered data loss prevention detecting sensitive data across SaaS, AI tools, and cloud. Increasingly focused on preventing data leakage through GenAI tools.

Elite security research and consulting firm with growing AI/ML security auditing practice. Develops open-source security tools and publishes ML security research.

Palo Alto Networks AI Security division - incumbent cybersecurity leader extending platform into AI-specific threat detection, model security, and AI-powered SOC automation.

AI research assistant for scientific research, automating literature reviews and data extraction across 138M+ papers. Public Benefit Corporation incubated at Ought, a nonprofit focused on scaling human reasoning.

AI-powered cybersecurity platform built by Unit 8200 veterans. Uses AI to enhance security operations - threat detection, response automation. $51.5M raised in 2025.

ML security platform that scans AI models and their dependencies for vulnerabilities - like a virus scanner for machine learning supply chains.

AI safety evaluation and compliance platform. Combines testing (red-teaming, safety evaluation) with compliance automation (documentation, audit trails). $30M raised in 2025.

AI security platform providing testing, red-teaming, and runtime protection for enterprise AI deployments. Works with government and defence clients on AI assurance.

AI security company that defends against prompt injection attacks, content filtering, and LLM firewalls. Protects AI applications in production from adversarial inputs.

AI evaluation and testing platform that helps companies verify their AI models work correctly before and after deployment. Automated testing for hallucinations, toxicity, and compliance.

AI governance and risk management platform that helps enterprises audit, monitor, and document their AI systems for regulatory compliance. Covers bias detection, risk assessment, and audit trails.

Dynamic SaaS security platform that discovers and monitors AI and SaaS applications across an enterprise. Helps companies track shadow AI and unauthorised SaaS usage.

Runtime application security and observability. Monitors applications at runtime to detect and prevent security threats including in AI-powered applications.

Pioneer of open-source image generation through Stable Diffusion. Facing governance challenges around content safety, CSAM risks, and financial sustainability.

Open-source framework for validating LLM outputs - checking for hallucinations, toxicity, PII leakage. Guardrails Hub offers community-contributed validators.

LLM evaluation and observability platform focused on hallucination detection, data quality scoring, and output monitoring for safer AI applications.

GenAI data security platform preventing sensitive data leakage through AI tools. Monitors and controls employee use of AI services, addressing shadow AI.

LLM testing and evaluation platform enabling CI/CD-style quality assurance for AI applications - regression testing, evaluation suites, and continuous monitoring.

Synthetic data platform generating privacy-preserving training data for computer vision. Eliminates need to collect real human images, reducing bias and privacy risks.

AI governance platform providing privacy-preserving AI, content moderation, and regulatory compliance guardrails for enterprise LLM deployments.

GenAI firewall monitoring and controlling all AI interactions within organizations. Protects against prompt injection, data leakage, and shadow AI usage.

Pioneers MLDR (Machine Learning Detection & Response) - runtime protection for AI models against adversarial attacks, model poisoning, and inference manipulation.

AI-powered software supply chain security firewall. Detects 4x more vulnerabilities than public sources. Discovered CVEs in TensorFlow and llama-cpp-python ML libraries.

AI compliance and verification tooling focused on EU AI Act requirements. Early-stage UK-based startup building compliance automation.

AI model validation and bias detection platform (formerly Fairly AI). Helps organisations test their AI models for fairness and compliance before deployment. Merged with Anch.ai (Jun 2025).

AI-powered biosecurity company that detects pathogens and assesses dual-use biological risks. Works at the intersection of AI capabilities and biological threat detection.

Builds adversarial testing tools that probe AI models for safety vulnerabilities - finding ways models can be manipulated before attackers do. CMU-affiliated. Halcyon portfolio.

AI security testing platform that finds vulnerabilities in AI models through automated adversarial testing. Think penetration testing, but for AI models instead of networks.

Makes AI models more interpretable and steerable - tools that let you see inside neural networks and adjust their behaviour. Founded by OpenAI and DeepMind alumni. $50M Series A backed by Anthropic, Lightspeed, and Halcyon.

Automated red-teaming platform that systematically probes AI models for safety vulnerabilities. Discovers failure modes that manual testing misses. Halcyon portfolio company.

Enterprise AI security with real-time observability, policy enforcement, and an AI verifier marketplace. Pre-seed stage.

Specialises in detecting whether AI models are being deceptive - specifically testing if models engage in scheming or strategic deception. The only company focused specifically on deceptive alignment evaluations.

The AI Underwriting Company - building AI-specific insurance products and the AIUC-1 safety standard for underwriting AI risk. First mover in AI insurance. Halcyon portfolio.

Builds adversarial testing tools that probe AI models for safety vulnerabilities - finding ways models can be manipulated before attackers do. CMU-affiliated. Halcyon portfolio.

Cryptographic verification and monitoring for AI infrastructure. Ensures AI compute and model execution hasn't been tampered with. Backed by Eric Schmidt. Halcyon portfolio.

AI security platform providing continuous security posture management for AI/ML systems. Identifies vulnerabilities, misconfigurations, and risks across AI model deployments.

AI privacy and security infrastructure - encryption for AI interactions. Protects data flowing through AI systems from interception and misuse. Halcyon portfolio.

AI-led compliance automation platform that handles security questionnaires and third-party risk management. Uses AI to automate the compliance documentation burden.

Data-in-use encryption using fully homomorphic encryption. Allows computation on encrypted data - relevant for AI processing sensitive data without exposure.

Non-human identity security - managing and securing machine identities, API keys, and service accounts. Critical infrastructure for AI agent authentication.

AI security assistant that automatically fixes vulnerabilities found by security scanners. Uses AI to write remediation code.

Open source vulnerability remediation. Automatically patches vulnerable open-source dependencies including those used in AI/ML pipelines.

Application Detection and Response (ADR) - monitors applications at runtime for security threats. Can detect attacks on AI applications as part of broader application security.

Web3 blockchain security with real-time crypto attack detection. Cyber Club London portfolio.

Cloud canary resources and detection - deploys decoy resources in cloud environments to detect attackers. Applicable to AI infrastructure security.

AI-native security platform protecting LLMs against adversarial attacks and model vulnerabilities. Israeli cybersecurity pedigree applied to AI/ML model security.

AI model security focused on trojan/backdoor detection and ML supply chain integrity. Scans models for malicious modifications before deployment.

Automated LLM security testing platform for prompt injection and conversational AI vulnerabilities. European (Croatian) AI security startup.

AI-powered deepfake detection for voice and video. Protects organizations against synthetic identity fraud and AI-generated impersonation.

GenAI security platform providing end-to-end protection for LLM applications from development through runtime. Detects prompt injection, data leakage, and model abuse.

AI supply chain security platform providing trust scoring and vulnerability scanning for AI models. Helps organizations assess security of third-party models.

AI evaluation platform providing standardized testing and benchmarking for AI system compliance with safety, fairness, and regulatory requirements.

AI application security platform providing protection for GenAI-powered enterprise applications. Founded by veteran cybersecurity leaders.

AI security platform providing automated red teaming and guardrails for LLM applications. Helps organizations identify safety risks through adversarial testing.

Confidential computing platform using trusted execution environments for privacy-preserving data processing. Implications for secure AI model training on sensitive data.

LLM security platform protecting enterprise GenAI deployments from prompt attacks and data leakage in real-time. Israeli cybersecurity pedigree.

AI security posture management providing visibility into AI/ML assets, risks, and governance across organizations. Discovers and monitors AI systems.

Adversarial AI research and red teaming platform. Maintains the AI Attack Surface Map - a comprehensive catalog of known AI attack vectors.

Agent infrastructure platform (formerly MemGPT) providing memory and state management for safe AI agent deployment. Tackles agent safety through controlled state management.

AI governance advisory firm providing policy and compliance consulting to organizations deploying AI systems. Helps navigate the regulatory landscape.

AI model validation and bias detection platform (formerly Fairly AI). Helps organisations test their AI models for fairness and compliance before deployment. Merged with Anch.ai (Jun 2025).

Specialized Language Models platform that trains and routes queries to task-specific models 1000x smaller than general-purpose LLMs. Achieves 85%+ cost reduction with customer-owned models.

German industrial AI startup applying computer vision and deep learning to non-destructive testing (NDT). Automates defect detection in industrial inspections for energy, manufacturing, and infrastructure sectors.

Runtime security platform for enterprise AI agents - like Snort for networks or Falco for containers, but for AI. 1,000+ threat signatures with 100% local processing and zero data transit. Built by UK Government, Mandiant, FireEye, and CrowdStrike veterans.

Model Evaluation and Threat Research - conducts frontier model capability evaluations to assess whether AI systems pose catastrophic risks. Works with governments and frontier labs to test dangerous capabilities.

UK-based nonprofit conducting AI safety evaluations and alignment research. Smaller evaluation org focused on specific safety testing methodologies.

Pure safety research lab led by Ilya Sutskever, co-founder of OpenAI. Raised $3B at $32B valuation with no product, no revenue, and no timeline. Betting everything on solving superintelligence alignment before building products.

Founded by former OpenAI CTO Mira Murati. Raised $2B at $12B valuation (July 2025). Frontier AI research with a safety focus. Not yet launched any products.

UK-based alignment research startup exploring cognitive emulation and control approaches to AI safety. Pivoted from earlier product ambitions to focus on fundamental research.

AI governance intelligence platform providing visibility, risk, and performance insights across enterprise AI usage. UK-based pre-seed startup.

AI safety research company. Halcyon portfolio. Limited public information available about specific research direction.

AI safety tooling focused on responsible scaling infrastructure - building tools that help AI labs scale safely. Halcyon portfolio.

Building formal verification tools for AI systems - mathematically proving safety properties of neural networks. Applies techniques from software verification to AI safety.

AI safety research focused on scalable oversight and alignment at the base model layer. Halcyon portfolio company exploring new approaches to keeping AI systems aligned as they scale.

AI safety and governance infrastructure startup. Halcyon portfolio company building governance tooling.

AI transparency research startup co-founded by Jacob Steinhardt (UC Berkeley). Working on interpretability methods to make AI model decisions more transparent. Halcyon portfolio.

AI safety startup incubated by Halcyon Futures. Pre-seed stage with minimal public information.

Research lab pioneering developmental interpretability using singular learning theory to understand neural network learning dynamics at a fundamental mathematical level.

AI agent security platform protecting autonomous AI systems from manipulation and unauthorized actions. Runtime defense for agentic AI deployments.

AI agent platform for robotics teams. LLM-powered code generation for hardware integrations, automating sensor integration and error triage. $1.6M raised.

Tracks and forecasts AI capabilities by analysing compute trends, training runs, and benchmark progress. The definitive source for 'how fast is AI progressing?' data.

Nonprofit alignment lab focused on AI control techniques and adversarial training. Produced foundational work on control protocols that frontier labs now reference in safety cases.

Centre for the Governance of AI - Oxford-based policy research organisation studying how AI should be governed. Research shapes international AI policy.

Open-source AI research collective that builds interpretability tools, evaluation frameworks, and open models. Created the widely-used Language Model Evaluation Harness.

Reduces societal-scale risks from AI through research and field-building. Created WMDP benchmark (widely used) and orchestrated the famous open letter on AI risk signed by hundreds of researchers.

Intensive research programs introducing researchers from other fields to AI safety. One of the earliest AI safety field-building organisations.

Runs AI safety research hackathons, sprints, and collaborative research events. Builds community and entry points for researchers new to AI safety.

Runs AI safety courses and bootcamps that train the next generation of AI safety professionals. The primary talent pipeline for the AI safety field - many researchers and founders in this ecosystem are BlueDot alumni.

Focuses on societal resilience in the era of transformative AI. Launch-granted by Halcyon Futures. Building frameworks for how society adapts to advanced AI.

Alignment Research Center - conducts alignment evaluation research, studies model organisms of misalignment, and works on eliciting latent knowledge from AI systems. Founded by Paul Christiano, one of the most influential alignment researchers.

Machine Intelligence Research Institute - the original AI alignment research organisation, founded in 2000. Focuses on mathematical foundations of alignment and agent foundations theory. Pioneer of the field.

Technical AI governance research organisation working on adversarial robustness, alignment evaluation, and responsible scaling policy. Bridges alignment research and governance policy.

Nonprofit philanthropic fund that researches, funds, and scales solutions to mitigate AI-enabled biosecurity risks. Two pillars: safeguarding powerful biological AI models and improving governance of synthetic nucleic acid synthesis.