|
1 | | -# Release Notes: LLM-Guard v0.4.1 |
| 1 | +# Release Notes: LLM-Guard v0.9.0 |
2 | 2 |
|
3 | | -> **AI Coding Hackathon Release** — Production-ready prompt injection detection with multi-provider LLM enrichment |
| 3 | +> **Release Focus** — Configurable input guardrail, hardened test coverage, refreshed rig workflows, and release tooling for the next tagged drop. |
4 | 4 |
|
5 | 5 | --- |
6 | 6 |
|
7 | 7 | ## 🎯 Overview |
8 | 8 |
|
9 | | -LLM-Guard v0.4.1 is a **fast, explainable Rust CLI** for detecting prompt injection and jailbreak attempts in LLM applications. This release delivers production-grade multi-provider LLM integration, enhanced detection rules, and comprehensive debug capabilities. |
| 9 | +LLM-Guard v0.9.0 elevates the original hackathon prototype into a polished developer tool. Highlights include: |
10 | 10 |
|
11 | | -**Developed in ~7 hours** during the [AI Coding Accelerator](https://maven.com/nila/ai-coding-accelerator) hackathon using AI-assisted development (GPT-5 Codex + Claude Code). |
| 11 | +- A **configurable input-size guardrail** applied consistently across stdin, file, and tail mode (default 1 MB, override via `--max-input-bytes` or `LLM_GUARD_MAX_INPUT_BYTES`). |
| 12 | +- **Expanded regression coverage** for tail-mode edge cases, input validation, and rig adapter fallbacks so providers that emit fenced or malformed JSON now fail safely. |
| 13 | +- **Operator assets** such as branded example prompts and a maintainer-oriented `docs/RELEASE_CHECKLIST.md`. |
| 14 | +- Documentation updates aligning with the current workspace (architecture, rule authoring, rig walkthrough) plus refreshed CLI integration tests demonstrating rig-backed providers end-to-end. |
12 | 15 |
|
13 | | ---- |
14 | | - |
15 | | -## ✨ Key Features |
16 | | - |
17 | | -### Core Capabilities |
18 | | -- ⚡ **Fast Scanning:** <100ms for typical prompts using Aho-Corasick + compiled regex |
19 | | -- 📊 **Transparent Risk Scoring:** 0-100 scale with detailed rule attribution and text excerpts |
20 | | -- 🔌 **Multi-Provider LLM Support:** OpenAI, Anthropic, Google Gemini, Azure OpenAI via `rig.rs` |
21 | | -- 🏥 **Provider Health Checks:** Built-in diagnostics for validating connectivity and configuration |
22 | | -- 📁 **Flexible Input Sources:** Files, stdin, streaming logs (tail mode) |
23 | | -- 📤 **Multiple Output Formats:** Human-readable CLI or JSON for CI/CD automation |
24 | | -- 🚦 **Exit Code Integration:** 0=low, 2=medium, 3=high, 1=error |
25 | | - |
26 | | -### Detection Coverage |
27 | | -- **Instruction Override:** `INSTR_IGNORE`, `INSTR_OVERRIDE` patterns |
28 | | -- **Data Exfiltration:** `PROMPT_LEAK` detection with flexible regex |
29 | | -- **Policy Subversion:** `MODEL_OVERRIDE` jailbreak patterns |
30 | | -- **Obfuscation Techniques:** `CODE_INJECTION` payload recognition |
| 16 | +🎬 **Human roles:** architect, tester, and product manager partnered with GPT-5 Codex and Claude Code throughout this release. |
31 | 17 |
|
32 | 18 | --- |
33 | 19 |
|
34 | | -## 🐛 Critical Fixes |
35 | | - |
36 | | -### Gemini Provider Integration |
37 | | -**Problem:** Rig.rs deserialization errors (`missing field generationConfig`) and API rejection of function calling with JSON MIME type |
38 | | -**Solution:** Bypassed rig entirely; implemented standalone HTTP client using Gemini's native REST API |
39 | | -**Impact:** Gemini now fully functional with `generationConfig.responseMimeType: "application/json"` |
40 | | - |
41 | | -### OpenAI GPT-5 Reasoning Models |
42 | | -**Problem:** Models returned only reasoning traces (no textual content) with `json_schema` response format |
43 | | -**Solution:** Switched from strict `json_schema` to flexible `json_object` format |
44 | | -**Impact:** Full compatibility with GPT-5 reasoning models; cleaner codebase |
45 | | - |
46 | | -### Detection Rules Gap |
47 | | -**Problem:** Keyword "ignore previous instructions" missed variations like "ignore **your** previous instructions" |
48 | | -**Solution:** Added flexible regex patterns `INSTR_IGNORE` and `PROMPT_LEAK` to `rules/patterns.json` |
49 | | -**Impact:** Scanner now catches attack variations; heuristic and LLM verdicts align |
50 | | - |
51 | | -**Example:** |
52 | | -``` |
53 | | -Before: Risk Score: 0.0 (Low), No findings |
54 | | -After: Risk Score: 37.5 (Medium) |
55 | | - Findings: PROMPT_LEAK [40.0], INSTR_IGNORE [35.0] |
56 | | -``` |
57 | | - |
58 | | -### Debug Logging Enhancement |
59 | | -**Problem:** `--debug` flag only logged errors, not all raw LLM responses |
60 | | -**Solution:** Added universal debug logging for all providers (raw response + extracted content) |
61 | | -**Impact:** Easier diagnosis of parsing issues and provider behavior quirks |
62 | | - |
63 | | ---- |
| 20 | +## ✨ What’s New |
64 | 21 |
|
65 | | -## 📦 What's Included |
| 22 | +### Guardrail & CLI Enhancements |
| 23 | +- `--max-input-bytes` CLI flag and `LLM_GUARD_MAX_INPUT_BYTES` env var expose the input-size limit (default 1 MB). |
| 24 | +- Tail mode streams through the same chunked UTF-8 reader, failing fast when the limit is exceeded and preserving snapshot integrity. |
| 25 | +- All health checks can now run in dry-run mode; OpenAI profiles are validated without live API calls (ignored on macOS sandbox due to SystemConfiguration limits). |
66 | 26 |
|
67 | | -### Binaries |
68 | | -```bash |
69 | | -# Build from source |
70 | | -cargo build --release |
71 | | -./target/release/llm-guard --version # v0.4.1 |
72 | | -``` |
| 27 | +### Test Hardenings |
| 28 | +- Added property/fuzz tests for tail behaviour, custom limits, and env/CLI precedence. |
| 29 | +- Rig adapter regression tests cover fenced JSON payloads and ensure “unknown” fallbacks are emitted instead of panics. |
| 30 | +- Keyword rule generation now filters out whitespace-only patterns, preventing spurious validation failures. |
73 | 31 |
|
74 | | -### Configuration Files |
75 | | -- `llm_providers.example.yaml` — Multi-provider config template |
76 | | -- `rules/keywords.txt` — Exact-match keyword database |
77 | | -- `rules/patterns.json` — Regex patterns for flexible detection |
| 32 | +### Documentation & DX |
| 33 | +- `docs/ARCHITECTURE.md` and `docs/RULE_AUTHORING.md` describe the workspace layout and rule pack conventions. |
| 34 | +- `docs/RELEASE_CHECKLIST.md` codifies release prep (fmt/clippy/test, docs, tagging, announcements). |
| 35 | +- `README.md` usage section reflects the guardrail flag, links to example prompts (`examples/prompt_safe.txt`, `prompt_suspicious.txt`, `prompt_malicious.txt`), and credits the human team roles. |
| 36 | +- `docs/USAGE.md` now documents the new flag/env var and showcases rig-backed workflows with health checks. |
78 | 37 |
|
79 | | -### Documentation |
80 | | -- `README.md` — Complete project overview with hackathon context |
81 | | -- `docs/USAGE.md` — Comprehensive CLI reference |
82 | | -- `docs/TESTING_GUIDE.md` — Testing protocols and provider health checks |
83 | | -- `AGENTS.md` — AI assistant onboarding guide |
84 | | -- `PLAN.md` — Implementation roadmap and phase tracking |
85 | | -- `PROJECT_SUMMARY.md` — Current state snapshot |
| 38 | +### Planning Updates |
| 39 | +- `PLAN.md` marks Phase 7–9 deliverables complete and defines the next steps: run the release checklist, groom stretch features (rule management, caching), and capture performance benchmarks. |
86 | 40 |
|
87 | 41 | --- |
88 | 42 |
|
89 | | -## 🚀 Quick Start |
| 43 | +## 🚀 Quick Start (unchanged) |
90 | 44 |
|
91 | | -### Installation |
92 | 45 | ```bash |
93 | 46 | git clone https://github.com/HendrikReh/llm-guard |
94 | 47 | cd llm-guard |
95 | 48 | cargo build --release |
96 | | -``` |
97 | | - |
98 | | -### Basic Usage |
99 | | -```bash |
100 | | -# Scan a file |
101 | | -./target/release/llm-guard scan --file examples/chat.txt |
102 | | - |
103 | | -# LLM-enhanced scan with Gemini |
104 | | -export LLM_GUARD_PROVIDER=gemini |
105 | | -export LLM_GUARD_API_KEY=your_key_here |
106 | | -./target/release/llm-guard scan --file examples/chat.txt --with-llm |
107 | | - |
108 | | -# Debug mode (dump raw responses) |
109 | | -./target/release/llm-guard scan --file examples/chat.txt --with-llm --debug |
110 | | - |
111 | | -# Provider health check |
112 | | -./target/release/llm-guard health --providers-config llm_providers.yaml |
113 | | -``` |
114 | | - |
115 | | -### CI/CD Integration |
116 | | -```bash |
117 | | -# Generate JSON output |
118 | | -./target/release/llm-guard scan --file input.txt --json > report.json |
119 | | - |
120 | | -# Exit codes: 0=low, 2=medium, 3=high, 1=error |
121 | | -if [ $? -ge 2 ]; then |
122 | | - echo "Security risk detected!" |
123 | | - exit 1 |
124 | | -fi |
125 | | -``` |
126 | 49 |
|
127 | | ---- |
| 50 | +# Scan with optional guardrail override |
| 51 | +./target/release/llm-guard-cli --max-input-bytes 2000000 scan --file examples/prompt_safe.txt |
128 | 52 |
|
129 | | -## 🔧 Configuration |
130 | | - |
131 | | -### Environment Variables |
132 | | -| Variable | Description | Default | |
133 | | -|----------|-------------|---------| |
134 | | -| `LLM_GUARD_PROVIDER` | Provider (`openai`, `anthropic`, `gemini`, `azure`) | `openai` | |
135 | | -| `LLM_GUARD_API_KEY` | API key/token | - | |
136 | | -| `LLM_GUARD_MODEL` | Model name (e.g., `gpt-4o-mini`) | Provider default | |
137 | | -| `LLM_GUARD_ENDPOINT` | Custom endpoint URL | Provider default | |
138 | | -| `LLM_GUARD_TIMEOUT_SECS` | HTTP timeout in seconds | `30` | |
139 | | -| `LLM_GUARD_MAX_RETRIES` | Retry count for failed calls | `2` | |
140 | | - |
141 | | -### Provider Profiles (`llm_providers.yaml`) |
142 | | -```yaml |
143 | | -providers: |
144 | | - - name: "openai" |
145 | | - api_key: "OPENAI_API_KEY" |
146 | | - model: "gpt-4o-mini" |
147 | | - - name: "gemini" |
148 | | - api_key: "GEMINI_API_KEY" |
149 | | - model: "gemini-1.5-flash" |
150 | | - - name: "azure" |
151 | | - api_key: "AZURE_OPENAI_KEY" |
152 | | - endpoint: "https://your-resource.openai.azure.com" |
153 | | - deployment: "gpt-4o-production" |
154 | | - api_version: "2024-02-15-preview" |
| 53 | +# Rig-backed scan with health pre-flight |
| 54 | +./target/release/llm-guard-cli --providers-config llm_providers.yaml health --provider openai --dry-run |
| 55 | +./target/release/llm-guard-cli scan --file examples/prompt_suspicious.txt --with-llm --provider openai |
155 | 56 | ``` |
156 | 57 |
|
157 | | -**Configuration Precedence:** CLI flags → Environment variables → Provider profile |
158 | | -
|
159 | | ---- |
160 | | -
|
161 | | -## 📊 Technical Metrics |
162 | | -
|
163 | | -| Metric | Value | |
164 | | -|--------|-------| |
165 | | -| **Lines of Code** | ~4,000 (Rust) | |
166 | | -| **Source Files** | 25 `.rs` files | |
167 | | -| **Test Coverage** | 44 tests (34 passing, 10 ignored) | |
168 | | -| **Dependencies** | Production-grade (tokio, reqwest, rig, clap) | |
169 | | -| **Detection Rules** | 4 patterns + keyword database | |
170 | | -| **Supported Providers** | 4 (OpenAI, Anthropic, Gemini, Azure) | |
171 | | -| **Performance** | <100ms for typical prompts | |
172 | | - |
173 | 58 | --- |
174 | 59 |
|
175 | 60 | ## 🧪 Testing |
176 | 61 |
|
177 | 62 | ```bash |
178 | | -# Run all tests |
| 63 | +cargo fmt |
| 64 | +cargo clippy --all-targets --all-features -- -D warnings |
179 | 65 | cargo test |
180 | | -
|
181 | | -# Run library tests only |
182 | | -cargo test --lib |
183 | | -
|
184 | | -# Run with ignored tests (requires network) |
| 66 | +# Include ignored provider tests when network access is available: |
185 | 67 | cargo test -- --include-ignored |
186 | | -
|
187 | | -# Provider health checks |
188 | | -cargo run --bin llm-guard-cli -- health --providers-config llm_providers.yaml |
189 | 68 | ``` |
190 | 69 |
|
191 | | -See [`docs/TESTING_GUIDE.md`](./docs/TESTING_GUIDE.md) for comprehensive testing protocols. |
192 | | - |
193 | | ---- |
194 | | - |
195 | | -## 🤖 AI-Assisted Development |
196 | | - |
197 | | -This release demonstrates the capabilities of **AI-assisted software development**: |
198 | | - |
199 | | -**Workflow:** |
200 | | -- **Primary Agent:** GPT-5 Codex (core logic, LLM adapters, CLI) |
201 | | -- **Review Agent:** Claude Code (code reviews, documentation, debugging) |
202 | | -- **Context Management:** RepoPrompt + Context7 MCP servers |
203 | | - |
204 | | -**What Worked:** |
205 | | -- ✅ Functional CLI with 4 LLM providers in <7 hours |
206 | | -- ✅ Multi-agent collaboration (coding vs. review separation) |
207 | | -- ✅ MCP integration eliminated manual file navigation |
208 | | -- ✅ PRD-driven development prevented scope creep |
209 | | - |
210 | | -**Challenges:** |
211 | | -- ⚠️ Provider API quirks (Gemini, OpenAI reasoning models) |
212 | | -- ⚠️ Testing gaps due to time pressure (10 ignored tests) |
213 | | -- ⚠️ Rig.rs limitations required Gemini bypass |
| 70 | +Tail fuzzer tests now run as part of the default suite; rig-related tests that require network/TLS remain ignored. |
214 | 71 |
|
215 | 72 | --- |
216 | 73 |
|
217 | | -## 🔮 Known Limitations |
| 74 | +## 📚 Documentation Snapshot |
218 | 75 |
|
219 | | -- **Rule Coverage:** Only 4 detection patterns (expandable via `rules/patterns.json`) |
220 | | -- **Context Windows:** Limited to 200-char proximity for synergy bonuses |
221 | | -- **Test Coverage:** 10 tests ignored (require network or specific environments) |
222 | | -- **Production Readiness:** Prototype for research/education; not audited for production security workloads |
| 76 | +| Document | Purpose | |
| 77 | +|----------|---------| |
| 78 | +| [`README.md`](./README.md) | Project overview, quick start, AI workflow insights | |
| 79 | +| [`docs/USAGE.md`](./docs/USAGE.md) | CLI reference, global flags, advanced scenarios | |
| 80 | +| [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md) | End-to-end data flow and component responsibilities | |
| 81 | +| [`docs/RULE_AUTHORING.md`](./docs/RULE_AUTHORING.md) | Extending keyword/regex packs safely | |
| 82 | +| [`docs/RELEASE_CHECKLIST.md`](./docs/RELEASE_CHECKLIST.md) | Maintainer tasks before tagging a release | |
| 83 | +| [`docs/TESTING_GUIDE.md`](./docs/TESTING_GUIDE.md) | Testing strategy, provider diagnostics | |
| 84 | +| [`docs/SECURITY.md`](./docs/SECURITY.md) | Guardrails and data-handling assumptions | |
223 | 85 |
|
224 | 86 | --- |
225 | 87 |
|
226 | | -## 📚 Resources |
| 88 | +## 📦 Packaging Checklist (Summary) |
227 | 89 |
|
228 | | -- **Main Documentation:** [README.md](./README.md) |
229 | | -- **Usage Reference:** [docs/USAGE.md](./docs/USAGE.md) |
230 | | -- **Testing Guide:** [docs/TESTING_GUIDE.md](./docs/TESTING_GUIDE.md) |
231 | | -- **Implementation Plan:** [PLAN.md](./PLAN.md) |
232 | | -- **AI Onboarding:** [AGENTS.md](./AGENTS.md) |
233 | | -- **Project Summary:** [PROJECT_SUMMARY.md](./PROJECT_SUMMARY.md) |
| 90 | +1. Run through `docs/RELEASE_CHECKLIST.md` (fmt/clippy/test, docs, version bumps). |
| 91 | +2. Tag `v0.9.0`, push tags, and draft a release with binaries + changelog. |
| 92 | +3. Share release summary with stakeholders (Slack/Discord/mailing list). |
234 | 93 |
|
235 | 94 | --- |
236 | 95 |
|
237 | | -## 🙏 Acknowledgments |
238 | | - |
239 | | -**Hackathon:** [AI Coding Accelerator](https://maven.com/nila/ai-coding-accelerator) (Maven) |
240 | | -**Instructors:** [Vignesh Mohankumar](https://x.com/vig_xyz), [Jason Liu](https://x.com/jxnlco) |
| 96 | +## 🔭 Next Steps |
241 | 97 |
|
242 | | -**Built with:** |
243 | | -- [Cursor](https://cursor.sh) + GPT-5 Codex |
244 | | -- [Claude Code](https://claude.ai) |
245 | | -- [RepoPrompt MCP](https://repoprompt.com/) |
246 | | -- [Context7 MCP](https://context7.com/) |
| 98 | +- Package the next tagged release using the new checklist. |
| 99 | +- Groom stretch features: rule management commands, verdict caching, tail streaming optimisations. |
| 100 | +- Capture performance benchmarks (latency, memory) across large prompts and document tuning guidance in `docs/ARCHITECTURE.md`. |
247 | 101 |
|
248 | 102 | --- |
249 | 103 |
|
250 | | -## 📄 License |
251 | | - |
252 | | -Apache-2.0 OR MIT |
253 | | - |
254 | | -**Security Disclaimer:** This tool is a prototype for research/education. Use at your own risk. |
255 | | - |
256 | | -**AI Development Notice:** Codebase primarily generated via AI assistants (GPT-5 Codex, Claude Code) with human oversight for architecture, testing, and quality validation. |
257 | | - |
258 | | ---- |
259 | | - |
260 | | -## 🔗 Links |
261 | | - |
262 | | -- **Repository:** https://github.com/HendrikReh/llm-guard |
263 | | -- **Issues:** https://github.com/HendrikReh/llm-guard/issues |
264 | | -- **Releases:** https://github.com/HendrikReh/llm-guard/releases |
265 | | - |
266 | | ---- |
| 104 | +## 📜 Historical Note — v0.4.1 |
267 | 105 |
|
268 | | -**Full Changelog:** https://github.com/HendrikReh/llm-guard/compare/v0.4.0...v0.4.1 |
| 106 | +The original hackathon release laid the foundation for LLM-Guard with multi-provider rig integration and core detection heuristics. Refer to the Git tag `v0.4.1` (or the GitHub Releases page) for the full historical changelog. |
0 commit comments