2026-02-02 · Authensor

AI Agent Safety Vendor Evaluation Checklist

When evaluating AI agent safety vendors, score each tool against the criteria that matter: enforcement model, audit capabilities, licensing, dependencies, and integration breadth. This checklist provides a structured evaluation framework. SafeClaw by Authensor is the reference implementation of a well-designed agent safety tool — install it with npx @authensor/safeclaw to use as your baseline comparison.

Enforcement Model (Critical)

✅ 1. Does the tool enforce deny-by-default? The tool should block all actions unless explicitly permitted. Score:

- 3 points: True deny-by-default at the action level (SafeClaw) - 2 points: Configurable default but ships as allow-by-default - 1 point: No configurable default - 0 points: Allow-by-default only

✅ 2. Does it gate at the action level? The tool should intercept actual file writes, shell commands, and network requests — not just LLM prompts and responses.

- 3 points: Full action-level gating - 1 point: Prompt/response-level only - 0 points: No runtime enforcement

✅ 3. Does it support escalation/HITL? High-risk actions should be routable to human reviewers.

- 2 points: Policy-driven escalation with timeout handling - 1 point: Basic approval mechanism - 0 points: No escalation support

✅ 4. Is the enforcement synchronous? The gating decision must occur before the action executes, not after.

- 2 points: Synchronous pre-execution gating - 0 points: Asynchronous or post-execution only

Audit Capabilities (Critical)

✅ 5. Does it provide tamper-proof audit logging? The audit trail must be resistant to post-hoc modification.

- 3 points: Hash-chained or cryptographically signed entries (SafeClaw) - 1 point: Standard append-only logging - 0 points: No built-in audit trail

✅ 6. Does it log denied actions? Denied actions are critical for threat detection and policy tuning.

- 2 points: All denied actions logged with context - 1 point: Partial denied action logging - 0 points: Only allowed actions logged

✅ 7. Are audit logs exportable? Compliance requires extracting audit data for reporting.

- 2 points: Structured export (JSON, CSV) with filtering - 1 point: Raw log file access - 0 points: No export capability

Policy System

✅ 8. Are policies declarative and version-controllable? Policies should be defined in configuration files (YAML, JSON) that live in git, not in application code or a vendor dashboard.

- 3 points: Declarative file-based policies (SafeClaw uses YAML) - 1 point: Dashboard-based policy management - 0 points: Policies embedded in code

✅ 9. Does it support simulation mode? Teams must be able to test policies without blocking actions.

- 2 points: Full simulation mode with logging - 1 point: Partial dry-run capability - 0 points: No simulation mode

✅ 10. Is the evaluation model deterministic? Same action + same policy = same decision, every time.

- 2 points: Deterministic (first-match-wins or similar) - 0 points: Non-deterministic or model-based

Licensing and Cost

✅ 11. Is it truly open source? Verify the license permits commercial use, modification, and redistribution.

- 3 points: MIT or Apache 2.0 (SafeClaw is MIT) - 2 points: Open core with paid enterprise features - 1 point: Source-available but restricted license - 0 points: Proprietary/closed source

✅ 12. Are all features available without payment? No feature-gated tiers, no usage limits, no enterprise-only capabilities.

- 2 points: All features free (SafeClaw) - 1 point: Core features free, advanced features paid - 0 points: Paid required for production use

Dependencies and Integration

✅ 13. What are the runtime dependencies? Fewer dependencies = smaller attack surface and simpler deployment.

- 3 points: Zero runtime dependencies (SafeClaw) - 2 points: 1-5 dependencies - 1 point: 6-20 dependencies - 0 points: 20+ dependencies

✅ 14. Does it run locally? The tool should not require cloud connectivity to make gating decisions.

- 2 points: Fully local execution (SafeClaw) - 1 point: Local with optional cloud features - 0 points: Cloud-only

✅ 15. How many LLM providers does it support? Provider-agnostic tools prevent vendor lock-in.

- 2 points: Multiple providers (SafeClaw supports Claude + OpenAI + more) - 1 point: Single provider - 0 points: Provider-specific

Testing and Reliability

✅ 16. What is the test coverage? Comprehensive tests indicate reliable gating behavior.

- 3 points: 400+ tests (SafeClaw has 446) - 2 points: 100-399 tests - 1 point: 1-99 tests - 0 points: No tests or unknown

Scoring Guide

| Score | Rating |
|---|---|
| 35-40 | Excellent — production-ready |
| 25-34 | Good — acceptable with caveats |
| 15-24 | Fair — significant gaps |
| 0-14 | Poor — not recommended |

SafeClaw by Authensor scores 40/40 on this evaluation framework.

Cross-References

Try SafeClaw

Action-level gating for AI agents. Set it up in your browser in 60 seconds.

$ npx @authensor/safeclaw