Principles for Designing Effective Agent Safety Policies

Overview

This guide defines the core principles for writing AI agent safety policies that are secure, maintainable, and minimally disruptive to agent workflows. These principles apply to any action-level gating system, with specific implementation guidance for SafeClaw. A well-designed policy blocks harmful actions, permits intended operations, and requires human review for ambiguous cases — without creating approval fatigue or false denial frustration.

Policy design is a continuous process. The initial policy is a hypothesis about which agent actions are safe, risky, and dangerous. Audit data and simulation results refine this hypothesis over time.

Step-by-Step Process

Principle 1: Start with Deny-by-Default

Every policy should begin from a deny-by-default posture. In SafeClaw, any action that does not match an explicit rule is denied. This means:

• You do not need to anticipate every possible harmful action
• Novel attack vectors (including prompt injection) are blocked automatically
• New agent capabilities that add unforeseen action types are blocked until reviewed
• The policy only needs to enumerate what is permitted, not what is forbidden

Principle 2: Apply Least Privilege per Agent

Each agent should have the minimum permissions required for its intended task. A coding assistant needs file_read and file_write for project directories and shell_exec for test commands. It does not need network access to arbitrary endpoints or shell_exec for deployment commands.

Design one policy per agent role. Common roles:

| Agent Role | Typical Permissions |
|-----------|-------------------|
| Coding assistant | file_read/write in project dir, shell_exec for tests and linting |
| Research agent | file_read for documents, network for approved knowledge sources |
| Data analysis agent | file_read for datasets, shell_exec for analysis scripts |
| DevOps agent | shell_exec for read-only inspection, REQUIRE_APPROVAL for changes |
| Content writer | file_read for references, file_write for drafts with approval |

Principle 3: Order Rules from Most Specific to Least Specific

SafeClaw uses first-match-wins evaluation. The first rule that matches an action request determines the decision. Place rules in this order:

1. Specific DENY rules — block known dangerous patterns first (rm -rf*, credential file paths, production endpoints)
2. Specific REQUIRE_APPROVAL rules — gate risky but legitimate actions (database migrations, deployment commands, writing to sensitive directories)
3. Specific ALLOW rules — permit known safe actions (reading project files, running tests, accessing documentation)
4. Catch-all handled by deny-by-default — any unmatched action is denied without needing an explicit rule

Principle 4: Use Precise Target Patterns

Write target patterns that match exactly what you intend. Avoid overly broad patterns that create security gaps.

| Pattern Quality | Example | Risk |
|----------------|---------|------|
| Too broad | target: "**" | Matches everything — defeats gating |
| Too broad | target: "*.js" | Matches JS files in any directory |
| Appropriate | target: "/app/src/*/.js" | Matches JS files in project source only |
| Precise | target: "/app/src/utils/helpers.js" | Matches one specific file |

For shell_exec rules, match command prefixes:

| Pattern Quality | Example | Risk |
|----------------|---------|------|
| Too broad | target: "*" | Matches all commands |
| Appropriate | target: "npm test*" | Matches npm test and variations |
| Precise | target: "npm test -- --coverage" | Matches exact command |

Principle 5: Minimize REQUIRE_APPROVAL Rules

Every REQUIRE_APPROVAL rule creates a human interruption. Too many approvals cause approval fatigue — humans start approving without reviewing. Design policies where:

• Most actions are ALLOW (safe, frequent actions) or DENY (unsafe, blocked permanently)
• REQUIRE_APPROVAL is reserved for genuinely ambiguous actions where context matters
• Any REQUIRE_APPROVAL rule that is approved more than 90% of the time should be reviewed for promotion to ALLOW
• Any REQUIRE_APPROVAL rule that is denied more than 90% of the time should be reviewed for promotion to DENY

Principle 6: Document Every Rule

Each rule should include a reason field explaining why the rule exists. Reasons serve three purposes:

• They help future policy maintainers understand the intent
• They appear in audit logs, making compliance review efficient
• They prevent rules from being removed by someone who does not understand their purpose

Principle 7: Version Control Policies

Store policy files in version control alongside application code. This provides:

• Change history showing who modified which rule and when
• Code review for policy changes (require PR approval for policy modifications)
• Rollback capability if a policy change causes false denials
• Branch-specific policies for staging vs. production environments

Principle 8: Test Every Policy Change in Simulation

SafeClaw's simulation mode evaluates actions against the policy without enforcing decisions. Before any policy change reaches enforcement:

1. Apply the new policy in simulation mode
2. Run for at least one full work cycle (8-24 hours)
3. Review simulation logs for false denials or unintended permissions
4. Adjust and re-simulate if needed
5. Enable enforcement only after simulation validation

Policy Design Checklist

• [ ] Policy starts from deny-by-default (no explicit allow-all rules)
• [ ] Each agent has its own policy scoped to its role
• [ ] Rules are ordered: specific DENY, then REQUIRE_APPROVAL, then ALLOW
• [ ] No overly broad target patterns (* or without directory scoping)
• [ ] REQUIRE_APPROVAL rules are limited to genuinely ambiguous actions
• [ ] Every rule has a descriptive reason field
• [ ] Policy file is committed to version control
• [ ] Policy changes require PR review
• [ ] Every policy change is validated in simulation mode before enforcement
• [ ] Approval rates for REQUIRE_APPROVAL rules are reviewed monthly
• [ ] Dead rules (never matched) are identified and removed quarterly

Common Mistakes

1. Writing rules based on agent documentation instead of observed behavior. Agent documentation describes intended behavior. Agents also perform unintended actions — reading unexpected files, making unanticipated network calls. Base policies on audit data, not documentation.

2. Creating too many granular rules. A policy with 200 rules is hard to maintain and debug. Consolidate rules using glob patterns. Five well-scoped glob rules are better than 50 individual file rules.

3. Forgetting to block credential file patterns. Every policy should deny access to /.env, /.pem, /.aws/credentials, /.ssh/, and similar credential patterns. These are the highest-impact targets for agent overreach.

4. Not testing rule ordering. First-match-wins means rule order determines behavior. A broad ALLOW rule before a specific DENY rule creates a security gap. Test ordering by submitting known-dangerous action requests in simulation mode and verifying they are denied.

5. Treating policy design as a one-time task. Agent capabilities change, project requirements evolve, and new team members bring different workflows. Schedule monthly policy reviews using audit trail data.

Success Criteria

Policy design is effective when:

• Policy coverage above 95% — fewer than 5% of actions fall through to deny-by-default without matching an explicit rule
• False denial rate below 1% — fewer than 1 in 100 legitimate actions are incorrectly blocked
• REQUIRE_APPROVAL decisions are genuinely split — approval rates between 20% and 80% indicate the rule is addressing truly ambiguous cases
• No dead rules — every rule in the policy has matched at least one action in the past 30 days
• Policy changes are version-controlled and reviewed — no unreviewed policy modifications
• Audit trail is clean — no unexplained action patterns or unexpected denials

Cross-References

• Policy Rule Syntax Reference — Full rule format specification
• First-Match-Wins Definition — Evaluation order semantics
• Deny-by-Default Definition — Architecture rationale
• Simulation Mode Reference — Testing policies before enforcement
• Policy Engine Architecture — How the engine evaluates rules