Metadata-Version: 2.4
Name: sentinelforge
Version: 0.1.0
Summary: SentinelForge: free evaluator kit for grading and gating AI agents (Sentinel, Forge, Scorecard, Baseline). BYO keys.
Author-email: Ultrec AI Inc <hello@ultrecai.com>
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# SentinelForge™ 0.1.0

A free evaluator kit from Ultrec AI Inc for **grading** and **gating** AI agents.
It is not a chat AI, and it does no weight training — it measures and gates the agents you already run.

| Component | What it does | Status |
|---|---|---|
| **Sentinel** | Independent LLM grader: compares an agent's answer to a key, flags claimed-but-not-run tools, reports agreement rates | Free (this package) |
| **Forge** | Skill gate: versions skill procedures, blocks promotion until tests pass, rolls back bad edits (SQLite) | Free (this package) |
| **Scorecard** | Builds scorecards from agent event logs: accuracy, per-category, Brier calibration, cost/tokens/time | Free (this package) |
| **Baseline** | Freezes seeded task sets with keys split out, hashes them, verifies them over time | Free (this package) |

## Install
```bash
unzip sentinelforge-0.1.0.zip && cd sentinelforge-0.1.0
pip install .            # Python 3.9+, no third-party dependencies
```

## Bring your own keys
Sentinel calls any OpenAI-compatible chat endpoint with **your** key. Nothing is sent anywhere unless you set one:
```bash
export SENTINELFORGE_API_KEY=sk-...                         # or OPENAI_API_KEY
export SENTINELFORGE_BASE_URL=https://api.openai.com/v1     # any OpenAI-compatible URL (Groq, OpenRouter, local...)
export SENTINELFORGE_MODEL=gpt-4o-mini
```

## Quickstart
```bash
sf --version
sf grade --task "What is 2+2?" --answer "4" --key "4"        # live grade (needs key); offline it says so and falls back to key-match
sf grade --events events.jsonl --session session.jsonl      # replay a run log, prints agreement
sf gate --db skills.db                                      # Forge gate report on a skills DB
sf gate --demo                                              # offline demo on an in-memory DB
sf scorecard --events events.jsonl --ai my-agent --out out/  # scorecard from an events log
sf baseline --demo                                          # offline freeze + verify demo
sf baseline --tasks tasks.json --sha <sha256>               # verify a frozen task set
```
Python:
```python
from sentinelforge import sentinel, forge, scorecard, baseline
sentinel.grade_one("What is 2+2?", "4", key="4")
```

## Run tests
`python -m unittest discover -s tests`

## License
MIT for all four modules (see LICENSE). © 2026 Ultrec AI Inc · ultrecai.com
