MODULE 25 — Commercial and Legal: Vendor Evaluation & AI Contracts¶
25.1 The Gap Most Architects Ignore¶
Technical architects are comfortable evaluating architecture. They are not comfortable evaluating contracts. This creates a predictable pattern: the architect recommends a vendor based on technical merit, the legal team reviews the contract without understanding the AI-specific risks, and the organization signs a contract that doesn't protect it.
The architect's role in the commercial process is not to be a lawyer. It is to ensure that the legal and procurement teams know what questions to ask and what terms to push for — because they do not know the AI-specific risks unless someone tells them.
25.2 The Vendor Evaluation Framework¶
Building the Evaluation Scorecard¶
Before sending an RFP or sitting in a demo, build a scoring framework. Vendors will optimize their presentation for whatever criteria you signal as important. If you don't specify criteria upfront, vendors will present what makes them look best — not what matters to you.
AI VENDOR EVALUATION SCORECARD
TECHNICAL CAPABILITY (40% weight)
Quality on your actual data (not their benchmark):
Score 1-5: Did you run their product against YOUR data and
YOUR representative queries? Not their demo data.
Production reliability evidence:
Score 1-5: Can they provide uptime SLA history?
Do they have references at comparable scale?
Failure mode transparency:
Score 1-5: Can they describe the most common ways their system
fails? Do they have published accuracy benchmarks?
Red flag: vendors who cannot describe failure modes.
Security and compliance posture:
Score 1-5: SOC2 Type II, ISO 27001, relevant industry certs?
Data residency options? Enterprise DPA available?
INTEGRATION AND OPERABILITY (25% weight)
Integration complexity:
Score 1-5: How much engineering effort to integrate?
Does it work with your existing stack?
Observability and monitoring:
Score 1-5: What telemetry does it expose?
Can you integrate with your existing observability stack?
Customization and extensibility:
Score 1-5: Can you customize for your domain?
Can you bring your own models, knowledge bases, prompts?
COMMERCIAL TERMS (20% weight)
Pricing model:
Score 1-5: Is pricing predictable? Does it scale linearly?
Are there usage caps or burst charges?
Contract flexibility:
Score 1-5: Annual vs. multi-year? Exit provisions?
What happens to your data if you leave?
Vendor stability:
Score 1-5: Funding status, customer base size, key person risk?
SUPPORT AND PARTNERSHIP (15% weight)
Implementation support:
Score 1-5: Do they have enterprise implementation experience?
Is there a customer success function?
Roadmap alignment:
Score 1-5: Does their product roadmap align with your needs?
Are they building what you need, or what their
last customer asked for?
Running the Vendor Demo¶
Most vendor demos are theater. The vendor has rehearsed scenarios that make their product look perfect. To get useful signal:
VENDOR DEMO PROTOCOL
1. Supply your own test scenarios (do not use theirs)
Before the demo, send the vendor 10 scenarios derived from
your actual use case. Tell them: "Please demo using these."
If they insist on showing their own scenarios first, note that.
It signals they don't perform as well on real customer problems.
2. Include failure scenarios in your test cases
At least 3 of your 10 scenarios should be edge cases or
scenarios where the system should escalate or decline.
The vendor's handling of failure is more informative than
their handling of success.
3. Ask them to demonstrate the failure
"Show me what happens when the AI is wrong."
"Show me what happens when a user asks something out of scope."
"Show me what happens when the retrieval doesn't find relevant content."
Vendors who can't demo failure modes haven't thought about production.
4. Ask about production customers
"Can you connect us with a customer who has been in production
for more than 6 months in a use case similar to ours?"
Vendors without production customers at your scale are a risk.
5. Ask the hard questions in the room
- "What does your training data usage policy say exactly?"
- "What is the deprecation notice period for your AI models?"
- "What is your SLA for accuracy? How do you define accuracy?"
- "If we find your AI is producing harmful outputs, what is
the remediation process and SLA?"
Watch the reaction to these questions as much as the answers.
A vendor that becomes defensive or evasive at standard procurement
questions is revealing something about their culture.
25.3 AI Contract Key Clauses¶
This is the section for your legal team. Print it, share it, use it to brief them before they review any AI vendor contract.
Clause 1: Training Data Use¶
What to look for: Does the vendor have the right to use your data (prompts, responses, uploaded content) to train or improve their models?
What good looks like:
"Provider shall not use Customer Data to train, improve, or develop Provider's AI models or any other machine learning models."
What bad looks like:
"By using the service, you grant us a license to use your content to improve our services." (This language, common in consumer terms, means your data trains their models)
What to negotiate: Enterprise contracts should explicitly prohibit training data use. If the vendor insists on improvement rights, negotiate for: anonymization before use, opt-out rights, and a prohibition on using identifiable customer data.
Clause 2: Data Retention¶
What to look for: How long does the vendor retain your prompts, responses, and uploaded content after processing?
What good looks like:
"Provider will not retain Customer Data for longer than 30 days after processing, except as required by applicable law. Audit logs will be retained for [period] and accessible to Customer upon request."
What bad looks like:
"We may retain aggregated, anonymized data indefinitely." (Anonymization may be reversible; "indefinitely" is not acceptable)
What to negotiate: Maximum retention period for prompts and responses. Explicit definition of "anonymized." Right to request deletion.
Clause 3: Model Version Notification¶
What to look for: What notice does the vendor provide before changing the model version that powers the service?
What good looks like:
"Provider will provide at least 90 days written notice before deprecating any model version. Provider will notify Customer of material behavioral changes to production models within 30 days of such changes."
What bad looks like:
Silence on this topic. (Most vendor contracts say nothing about model versioning)
What to negotiate: Minimum deprecation notice period (90 days minimum for enterprise). Notification of material behavioral changes. The right to remain on a prior model version during a transition period.
Clause 4: Accuracy and Quality SLA¶
What to look for: Does the vendor make any commitments about the quality or accuracy of AI outputs?
Reality: Most AI vendors do not offer accuracy SLAs because output quality is context-dependent. Be skeptical of any vendor who claims a specific accuracy guarantee — it is likely based on a benchmark that doesn't match your use case.
What to negotiate: Not an accuracy SLA (which is usually meaningless), but a process SLA: "If Customer identifies systematic quality degradation, Provider will investigate and provide root cause analysis within 5 business days." The commitment is to the investigation process, not the output quality.
Clause 5: Security Incident Notification¶
What to look for: What are the vendor's obligations if a security incident exposes customer data?
What good looks like:
"Provider will notify Customer of any Security Incident within 72 hours of discovery. Notification will include: nature of the incident, categories and approximate number of individuals affected, likely consequences, and measures taken or proposed to address the incident."
This must be specific. Generic "we will notify you promptly" language does not satisfy GDPR Article 33 requirements.
What to negotiate: Specific notification timeline (72 hours is the GDPR standard). Required content of the notification. Cooperation obligations (providing information needed for regulatory reporting).
Clause 6: Audit Rights¶
What to look for: Can you audit the vendor's compliance with their data handling commitments?
What good looks like:
"Upon reasonable notice, Customer may audit Provider's compliance with this Agreement, including data handling practices, either directly or through an independent third party."
What bad looks like:
Absence of any audit rights. "Provider will provide a copy of its SOC2 report upon request." (A SOC2 report is not an audit of your specific data handling — it is a general security audit)
Particularly important for regulated industries. SR 11-7 successor guidance requires that model risk validation extend to third-party vendors. Without audit rights, this requirement cannot be met.
Clause 7: Exit and Data Portability¶
What to look for: What happens to your data and your customizations when you leave?
What good looks like:
"Upon termination, Provider will make Customer Data available for export in [standard format] for 90 days. Provider will certify deletion of Customer Data within 30 days of export completion."
What to negotiate: - Format of exported data (your fine-tuned model weights, your knowledge base content, your prompt history) - Transition assistance period - Reference customer obligations (some vendors require you to remain a reference customer as a condition of certain contract terms)
Clause 8: EU AI Act and Regulatory Compliance¶
For vendors providing AI used in high-risk applications, include:
"Provider represents and warrants that Provider's AI System complies with applicable AI regulations, including the EU AI Act where applicable. Provider will provide Customer with documentation sufficient for Customer to demonstrate compliance with applicable regulations, including technical documentation as required by Article 11 of the EU AI Act."
Why this matters: Many vendors are not yet compliant with EU AI Act high-risk requirements. Getting a representation that they are — and a documentation commitment — creates a contractual basis for remedies if compliance turns out to be absent.
25.4 The Due Diligence Checklist¶
Send this to the vendor as part of the evaluation process. Review the responses before contract negotiation.
AI VENDOR DUE DILIGENCE QUESTIONNAIRE
DATA HANDLING
1. What data do you collect from customer interactions?
(prompts, responses, user identifiers, metadata)
2. How long is this data retained?
3. Is customer data used to train or improve your models?
If yes: describe the process, anonymization approach, opt-out mechanism.
4. Where is data processed and stored? List all regions/countries.
5. What third-party processors have access to customer data?
SECURITY
6. What security certifications do you hold? (SOC2 Type II, ISO 27001)
Provide the most recent reports.
7. What is your process when a security incident occurs?
What is the notification timeline and content?
8. Describe your approach to prompt injection prevention.
9. How are model outputs filtered for harmful content?
AI-SPECIFIC
10. What is your policy for model version deprecation?
How much notice do you provide?
11. How do you notify customers of material behavioral changes
to production models?
12. What accuracy or quality benchmarks do you publish?
Are these independently validated?
13. How do you handle cases where your AI produces incorrect outputs?
What is the remediation process?
14. Describe the most common failure modes of your system.
(If they cannot answer this, flag it as a risk.)
COMPLIANCE
15. Have you completed an EU AI Act risk assessment for your system?
16. What documentation can you provide to support customer compliance
obligations (GDPR, EU AI Act, SR 11-7, HIPAA as applicable)?
17. Do you support customer audit rights? Describe the process.
COMMERCIAL
18. What is the full pricing model including overage charges?
19. What are the contract minimum terms and exit provisions?
20. What is the data export format and process upon termination?
EXERCISE — Contract Clause Analysis: Obtain the standard terms of service for any AI vendor you currently use or are evaluating. Find the clauses related to training data use, data retention, and model version changes. Does the contract address each of the eight clauses in Section 25.3? For each that is missing or inadequate: write the replacement clause language you would request.
PONDER — The Unsigned Protection: For AI vendors your organization currently uses: have you reviewed the training data use clause? Is your data being used to train their models? If you don't know, how would you find out?
WORKSHOP — Vendor Scorecard Build: For an AI vendor evaluation your organization is currently conducting or planning: build the complete evaluation scorecard from Section 25.2. Define the weights for your specific context. Design 10 demo scenarios using your actual use cases. Run the due diligence questionnaire. Score the vendor. Present the recommendation with explicit rationale for each scored dimension.
Next: Module 26 — AI Requirements Engineering