AI Vendor Risk Assessment: Evaluating Third-Party AI Providers

TL;DR — AI vendor risk assessment requires evaluating five domains: (1) security foundations (SOC 2, ISO 27001, penetration testing), (2) model transparency (training data sources, AIBOM, bias testing), (3) data governance (inference data retention, zero-retention policies, GDPR lawful basis), (4) regulatory compliance (EU AI Act Article 26, NIST AI RMF GOVERN 6), and (5) operational risk (model change notification, incident response, business continuity). 64% of organizations lack visibility into AI risk exposure. Fourth-party AI risk — when your vendor uses OpenAI or Anthropic — is the biggest blind spot. Require AIBOM disclosure, fourth-party sub-processor notice, and contractual protections for model behavior changes. The NIST AI RMF GOVERN 6 and ISO 42001 Clause 8.4 provide the compliance grounding.

Every SaaS vendor is now an AI vendor. Your CRM embeds AI for lead scoring. Your ITSM tool uses LLMs for ticket summarization. Your collaboration platform has an AI assistant. Each of these vendors processes your data through AI models you didn't build, didn't train, and don't control.

Traditional third-party risk management (TPRM) was not designed for this. A 2025 BigID report found that 64% of organizations lack visibility into AI risk exposure, and nearly half have no AI-specific security controls in place. The NIST AI RMF GOVERN 6 explicitly requires policies for "AI risks and benefits arising from third-party software, data, and other supply chain issues" — but most TPRM programs haven't caught up.

The uncomfortable truth: regulatory liability does not transfer to the vendor. EU AI Act Article 26 makes clear that deployers of high-risk AI systems bear obligations around human oversight, input data quality, monitoring, and record-keeping — even when the AI system was built by a vendor. If your vendor's AI processes personal data in violation of GDPR, you bear enforcement risk alongside the provider.

What Makes AI Vendors a Distinct Risk Category?

The biggest gap is fourth-party AI risk. When your SaaS vendor embeds OpenAI, Anthropic, or another model provider, you have a contract with the SaaS vendor — not the model provider. The vendor often cannot give you contractual guarantees about the model provider's data handling, model behavior, or security posture. You are exposed to a party you have no contractual relationship with.

What Are the Five Risk Domains?

Domain 1: Security Foundations

Start with traditional security, then layer AI-specific requirements.

Domain 2: Model Transparency

AI vendors must disclose how their models work, what they were trained on, and how they change over time.

aibom_requirements:
  model_architecture: "Required: model type, parameters, context window"
  training_data_sources: "Required: categories of training data, provenance"
  evaluation_results: "Required: benchmark scores, bias test results"
  known_limitations: "Required: documented biases, failure modes"
  model_version: "Required: current version, update history"
  dependencies: "Required: foundation model provider, libraries"
  hyperparameters: "Recommended: training configuration"
  data_lineage: "Recommended: data processing pipeline"

Key questions:
- What data was used for training? Can you prove valid consent for personal data?
- What are the known biases and limitations?
- How often is the model retrained? Do we get notified of behavioral changes?
- Can you explain why the model made a specific decision?

Domain 3: Data Governance

Domain 4: Regulatory Compliance

The vendor must align with the regulatory frameworks that apply to your deployment:

Domain 5: Operational Risk

How to Assess an AI Vendor: The 20-Question Security Questionnaire

from dataclasses import dataclass
from enum import Enum

class RiskLevel(Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"

@dataclass
class VendorAssessment:
    vendor_name: str
    questions: list[dict]
    scores: dict[str, RiskLevel]

    def assess(self) -> dict:
        """Run the 20-question AI vendor assessment."""
        results = {
            "vendor": self.vendor_name,
            "domains": {},
            "overall_risk": RiskLevel.LOW,
            "red_flags": [],
        }

        for q in self.questions:
            domain = q["domain"]
            answer = q.get("answer", "unknown")
            risk = self._score_answer(q, answer)

            if domain not in results["domains"]:
                results["domains"][domain] = []
            results["domains"][domain].append({
                "question": q["question"],
                "risk": risk,
            })

            if risk == RiskLevel.CRITICAL:
                results["red_flags"].append(q["question"])

        results["overall_risk"] = self._calculate_overall_risk(results)
        return results

    def _score_answer(self, question: dict, answer: str) -> RiskLevel:
        """Score an individual answer."""
        if answer == "unknown":
            return RiskLevel.CRITICAL
        if "no" in answer.lower() and question.get("required", True):
            return RiskLevel.HIGH
        return RiskLevel.LOW

    def _calculate_overall_risk(self, results: dict) -> RiskLevel:
        """Calculate overall risk from domain scores."""
        if results["red_flags"]:
            return RiskLevel.CRITICAL
        # Check for any HIGH risk answers
        for domain_answers in results["domains"].values():
            for a in domain_answers:
                if a["risk"] == RiskLevel.HIGH:
                    return RiskLevel.HIGH
        return RiskLevel.LOW

The 20 Questions

Security Foundations (1-5):
1. Do you hold a current SOC 2 Type II report covering AI systems? (Require evidence, not claims)
2. Do you conduct annual penetration tests that include AI-specific attack vectors (prompt injection, model extraction)?
3. Do you have a vulnerability disclosure program for AI components?
4. How are model weights protected against unauthorized access?
5. What access controls prevent unauthorized model manipulation?

Model Transparency (6-10):
6. Can you provide an AI Bill of Materials (AIBOM) on request?
7. What data sources were used to train the model? Can you prove lawful basis for personal data?
8. What are the known biases and limitations of your model?
9. How often is the model retrained, and do you notify customers of behavioral changes?
10. Can you explain why the model made a specific decision for a specific input?

Data Governance (11-15):
11. Do you retain customer inference data? If so, for how long and for what purpose?
12. Is customer data used to train or fine-tune your models?
13. Where is inference data processed, and can you guarantee data residency?
14. What is your GDPR lawful basis for processing personal data in AI inference?
15. Can we request deletion of our data from your training datasets?

Regulatory & Operational (16-20):
16. How do you classify your AI system under the EU AI Act (prohibited/high-risk/limited/minimal)?
17. Do you disclose all AI sub-processors (fourth-party providers like OpenAI, Anthropic)?
18. What is your incident notification timeline for AI-specific security incidents?
19. Do you align with NIST AI RMF or ISO 42001? Can you provide evidence?
20. What is your business continuity plan for AI service disruptions?

What Contractual Protections Are Required?

Standard software procurement agreements are insufficient for AI vendors. The following clauses must be added:

ai_vendor_contract_clauses:
  model_change_notification:
    requirement: "Vendor must notify customer of material behavioral changes"
    timeline: "30 days before deployment for planned changes, 24 hours for emergency"
    threshold: "Defined metrics for what constitutes 'material' change"

  aibom_delivery:
    requirement: "Vendor provides current AIBOM on request"
    timeline: "Within 30 days of request"
    updates: "Updated AIBOM within 30 days of model version change"

  fourth_party_disclosure:
    requirement: "Vendor discloses all AI sub-processors"
    notice: "60 days before adding or changing sub-processors"
    right_to_object: "Customer can object to new sub-processors"

  zero_retention:
    requirement: "Vendor does not retain customer inference data"
    scope: "Prompts, outputs, and metadata"
    exception: "Only with explicit written consent for each use case"

  right_to_audit:
    requirement: "Customer can audit vendor AI systems"
    method: "Directly or through independent third party"
    frequency: "Annually or upon incident trigger"
    scope: "Model performance, bias testing, data governance"

  data_provenance_warranty:
    requirement: "Vendor warrants training data was lawfully obtained"
    indemnity: "Vendor indemnifies against IP and privacy claims"
    scope: "Training data, fine-tuning data, evaluation data"

  incident_notification:
    requirement: "Vendor notifies customer of AI-specific incidents"
    timeline: "24 hours for S1/S2 incidents, 72 hours for S3"
    scope: "Prompt injection, data leak, model misuse, bias detected"

How to Monitor AI Vendors After Onboarding?

Assessment is not a one-time activity. AI vendors require continuous monitoring because model behavior changes, sub-processors are added, and new risks emerge.

Monitoring Triggers

Continuous Monitoring Architecture

Shadow AI detection: SaaS vendors silently add AI features to products you've already onboarded. This creates risk without your knowledge. Address this with continuous AI inventory monitoring — scan vendor release notes, API documentation, and product updates for new AI features, not just point-in-time assessments during procurement.

How to Align with NIST AI RMF and ISO 42001?

The NIST AI RMF GOVERN 6 and ISO 42001 Clause 8.4 both require third-party AI risk management:

The Treasury Department's Financial Services AI Risk Management Framework (February 2026) includes 230 control objectives mapped across the AI lifecycle, with third-party risk explicitly addressed. Financial services organizations should treat it as the applied version of NIST AI RMF principles.

AI Vendor Risk Assessment Checklist

• [ ] AI vendor inventory maintained with current status and risk tier
• [ ] Five-domain assessment completed before onboarding (security, transparency, data, regulatory, operational)
• [ ] 20-question security questionnaire completed with evidence (not self-attestation)
• [ ] SOC 2 Type II report obtained and verified (less than 12 months old)
• [ ] AIBOM requested and received from vendor
• [ ] Training data provenance verified (lawful basis for personal data)
• [ ] Fourth-party AI sub-processors disclosed
• [ ] Zero-retention policy for inference data confirmed in contract
• [ ] Model change notification clause in contract
• [ ] Right to audit AI systems in contract
• [ ] Data provenance warranty and indemnity in contract
• [ ] AI-specific incident notification timeline in contract (24h for S1/S2)
• [ ] Bias testing results requested and reviewed
• [ ] EU AI Act classification verified (prohibited/high-risk/limited/minimal)
• [ ] GDPR Article 28 DPA with AI-specific clauses executed
• [ ] Continuous monitoring triggers defined and implemented
• [ ] Shadow AI detection process for existing vendors
• [ ] Annual reassessment scheduled
• [ ] Vendor exit plan documented (data portability, model export)
• [ ] Integration with AI incident response plan

FAQ

What is AI vendor risk assessment?

AI vendor risk assessment is the process of evaluating third-party AI providers for security, data governance, model transparency, regulatory compliance, and operational risk before and during engagement. Unlike traditional vendor assessments, it covers AI-specific concerns: training data provenance, model behavior changes, fourth-party sub-processors (e.g., your SaaS vendor uses OpenAI), and AI Bill of Materials (AIBOM) disclosure.

How is AI vendor assessment different from traditional TPRM?

Traditional TPRM evaluates uptime, data security, and SLAs. AI vendor assessment must additionally evaluate model training data sources, bias testing results, model change notification policies, fourth-party AI sub-processor disclosure, inference data retention, and AI-specific security (prompt injection resistance, model extraction protection). Standard questionnaires like the SIG or ISO 27001 were not designed to cover these dimensions.

What is an AI Bill of Materials (AIBOM)?

An AI Bill of Materials (AIBOM) is a structured inventory of all components in an AI system: model architecture, training data sources, datasets used, hyperparameters, evaluation results, known biases, and dependencies. It is the AI equivalent of a Software Bill of Materials (SBOM). Require AI vendors to provide a current AIBOM on request and within a defined period (30 days is reasonable) for ongoing supply chain monitoring.

What is fourth-party AI risk?

Fourth-party AI risk occurs when your SaaS vendor embeds a third-party AI provider (like OpenAI, Anthropic, or Google) into their product. You have a contract with the SaaS vendor, not the model provider. The SaaS vendor often cannot give contractual guarantees about the model provider's data handling, model behavior, or security posture. You must require disclosure of all AI sub-processors and notification before adding or changing them.

What contractual protections should AI vendor agreements include?

AI vendor contracts must include: model change notification (behavioral changes, not just version updates), AIBOM delivery on request, fourth-party sub-processor disclosure and notice, zero-retention policy for inference data, right to audit AI systems, data provenance warranties, bias testing disclosure, and incident notification with specific AI-incident timelines. Standard software procurement agreements rarely contain these provisions.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →