Skip to content
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
da76127
Adding support for risk scenarios
thatfield1 Aug 16, 2025
21f105f
Move reusable descriptions to constants file
garrett-mccutcheon Sep 23, 2025
6c1616e
Add additional GET endpoints for Controls
garrett-mccutcheon Sep 23, 2025
5d39b57
Expand coverage for Frameworks and Tests endpoints
garrett-mccutcheon Sep 23, 2025
583477d
Add support for Integrations and Vendors
garrett-mccutcheon Sep 23, 2025
68054a8
Add integrations and vendors operations files I neglected to commit
garrett-mccutcheon Sep 23, 2025
b932911
Add support for Documents endpoints and relocate global-descriptions
garrett-mccutcheon Sep 23, 2025
9ae8d09
Add support for Policies endpoints
garrett-mccutcheon Sep 23, 2025
86abc64
Add discovered vendors operations and update eval system
garrett-mccutcheon Sep 23, 2025
f7712a9
Add support for Groups and People endpoints
garrett-mccutcheon Sep 23, 2025
05babe3
Add support for Vulnerabilities endpoints
garrett-mccutcheon Sep 23, 2025
fa673c8
Add support for Monitored Computers and Vendor Risk Attributes endpoints
garrett-mccutcheon Sep 23, 2025
409cd3f
Add support for Trust Centers endpoints
garrett-mccutcheon Sep 23, 2025
176a2b0
Update controls to use centralized descriptions
garrett-mccutcheon Sep 23, 2025
95581dd
Update code formatting with Prettier
garrett-mccutcheon Sep 24, 2025
32867e4
Refactor tool names and descriptions for clarity
garrett-mccutcheon Sep 24, 2025
00ff713
refactor: add comprehensive get endpoints and improve DRY patterns
garrett-mccutcheon Sep 24, 2025
4d5f46b
Add missing Trust Center GET endpoints
garrett-mccutcheon Sep 24, 2025
7393b80
Refactor to use an automated tool registry
garrett-mccutcheon Sep 24, 2025
72a6ffc
Refactor to a barrel import structure for consistency
garrett-mccutcheon Sep 24, 2025
fc63d3d
Refactor to consolidate get and list tools where possible
garrett-mccutcheon Sep 25, 2025
8013331
Perform more consolidation to bring the tool count under control
garrett-mccutcheon Sep 26, 2025
0d7dd07
Config file added to control which specific tools are enabled
garrett-mccutcheon Oct 3, 2025
9360778
Add and clarify optional parameters for tools
garrett-mccutcheon Oct 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -139,4 +139,10 @@ node_modules/
build/

# Claude Code generated files
CLAUDE.md
CLAUDE.md

# backup files
*.bak*

# Mac filesystem
*.DS_store
383 changes: 359 additions & 24 deletions README.md

Large diffs are not rendered by default.

73 changes: 58 additions & 15 deletions src/eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,19 +40,62 @@ OPENAI_API_KEY="your_openai_api_key_here" node build/eval/eval.js

## Test Cases

The evaluation includes 11 test cases covering:
The evaluation includes 66 test cases covering:

### ✅ **Tool Selection Tests**

- **AWS Security Review**: `get_tests` with AWS and NEEDS_ATTENTION filters
- **SOC2 Compliance**: `get_tests` with SOC2 framework filter
- **Entity Details**: `get_test_entities` for specific failing resources
- **Maintenance Deactivation**: `deactivate_test_entity` for suppressing alerts
- **Framework Listing**: `get_frameworks` for available frameworks
- **Control Requirements**: `get_framework_controls` for specific framework details
- **Status Percentage**: `get_frameworks` for completion percentages
- **Control Listing**: `get_controls` for all security controls
- **Control Tests**: `get_control_tests` for tests validating specific controls
- **AWS Security Review**: `list_tests` with AWS and NEEDS_ATTENTION filters
- **SOC2 Compliance**: `list_tests` with SOC2 framework filter
- **Entity Details**: `list_test_entities` for specific failing resources
- **Framework Listing**: `list_frameworks` for available frameworks
- **Control Requirements**: `list_framework_controls` for specific framework details
- **Status Percentage**: `list_frameworks` for completion percentages
- **Control Listing**: `list_controls` for all security controls
- **Control Tests**: `list_control_tests` for tests validating specific controls
- **Library Controls**: `list_library_controls` for available Vanta library controls
- **Control Documents**: `list_control_documents` for documents associated with controls
- **Control Details**: `get_control` for specific control information
- **Framework Details**: `get_framework` for specific framework information
- **Risk Details**: `get_risk` for specific risk scenario information
- **Integration Listing**: `get_integrations` for connected integrations
- **Integration Details**: `get_integration_by_id` for specific integration information
- **Vendor Listing**: `get_vendors` for all vendors
- **Vendor Details**: `get_vendor_by_id` for specific vendor information
- **Document Listing**: `get_documents` for all compliance documents
- **Document Details**: `get_document_by_id` for specific document information
- **Document Controls**: `get_document_controls` for controls associated with documents
- **Document Links**: `get_document_links` for external references in documents
- **Document Uploads**: `get_document_uploads` for file uploads attached to documents
- **Document Downloads**: `download_document_file` for intelligently downloading files (text content for readable files, metadata for binary files)
- **Policy Listing**: `get_policies` for all organizational policies
- **Policy Details**: `get_policy_by_id` for specific policy information
- **Discovered Vendors**: `get_discovered_vendors` for automatically discovered vendors
- **Discovered Vendor Accounts**: `get_discovered_vendor_accounts` for detailed vendor account information
- **Group Listing**: `get_groups` for all organizational groups
- **Group Details**: `get_group_by_id` for specific group information
- **Group Membership**: `get_group_people` for people in specific groups
- **People Listing**: `get_people` for all people in the organization
- **Person Details**: `get_person_by_id` for specific person information
- **Vulnerability Listing**: `get_vulnerabilities` for all detected vulnerabilities
- **Vulnerability Details**: `get_vulnerability_by_id` for specific vulnerability information
- **Vulnerability Remediations**: `get_vulnerability_remediations` for tracking remediation efforts
- **Vulnerable Assets**: `get_vulnerable_assets` for assets affected by vulnerabilities
- **Vulnerable Asset Details**: `get_vulnerable_asset_by_id` for specific asset vulnerability information
- **Monitored Computers**: `get_monitored_computers` for all computers being monitored for compliance
- **Computer Details**: `get_monitored_computer_by_id` for specific computer information
- **Vendor Risk Attributes**: `get_vendor_risk_attributes` for available risk assessment criteria
- **Trust Center Configuration**: `get_trust_center` for Trust Center settings and branding
- **Trust Center Access Requests**: `get_trust_center_access_requests` for managing customer access
- **Access Request Details**: `get_trust_center_access_request` for individual request information
- **Trust Center Analytics**: `get_trust_center_viewer_activity_events` for engagement tracking
- **Control Categories**: `get_trust_center_control_categories` for compliance organization
- **Category Details**: `get_trust_center_control_category` for specific category information
- **Published Controls**: `get_trust_center_controls` for public compliance controls
- **Control Details**: `get_trust_center_control` for specific control implementation
- **Trust Center FAQs**: `get_trust_center_faqs` for customer information
- **FAQ Details**: `get_trust_center_faq` for specific FAQ content
- **Trust Center Resources**: `get_trust_center_resources` for downloadable materials
- **Resource Documents**: `get_trust_center_document` for specific document details

### ❌ **Negative Tests**

Expand All @@ -65,10 +108,10 @@ The evaluation includes 11 test cases covering:
🧪 Vanta MCP Server Tool Evaluation
====================================

📝 Test: Should call get_tests with AWS filter and NEEDS_ATTENTION status
📝 Test: Should call list_tests with AWS filter and NEEDS_ATTENTION status
💬 Prompt: "What security issues do I have in my AWS infrastructure?"
🎯 Expected Tool: get_tests
✅ PASS: Correctly called get_tests
🎯 Expected Tool: list_tests
✅ PASS: Correctly called list_tests
✅ Parameters match expected values
📋 Called with: {
"statusFilter": "NEEDS_ATTENTION",
Expand All @@ -77,8 +120,8 @@ The evaluation includes 11 test cases covering:

📊 Final Results
================
✅ Passed: 11/11 tests
❌ Failed: 0/11 tests
✅ Passed: 66/66 tests
❌ Failed: 0/66 tests
📈 Success Rate: 100%
🎉 All tests passed! Tool calling behavior is working correctly.
```
Expand Down
Loading