How to Test and Evaluate AI Agents
Build a small agent evaluation dataset. Compare task success, tool correctness, permission outcomes, local timing and unknown cost with a Python fixture.
Ideas you can put to work
Explore software, AI, and the systems behind everyday applications.
Practical explanations, examples, and things to try.
Build a small agent evaluation dataset. Compare task success, tool correctness, permission outcomes, local timing and unknown cost with a Python fixture.
Separate read, draft and send permissions. Run a mock email workflow with exact-action approval, changed arguments, expiry, reuse checks and audit records.
Trace a harmless document injection through tool restrictions, output checks and mock-action approval. Run a local Python fixture and learn its limits.
Compare one assistant with a planner, researcher and reviewer. Run an offline Python example and evaluate contracts, failures, latency and token costs.
Build an inspectable context with instructions, evidence, tool results and recent state. Test a byte budget, reviewed excerpts and missing-evidence handoffs.
Compare product-document retrieval with saved user preferences. Run a support fixture that checks access, freshness, provenance and retrieval failures.
Separate model context, task state and saved preferences. Use Python and SQLite to test consent, scoped retrieval, stale updates, expiry and deletion.
Build a bounded support assistant with one read-only tool, a reviewable trace and clear handoffs. Run fixtures first, then connect an optional model.
Build a read-only HTTP adapter with validated inputs, controlled errors and bounded retries. Test a local API without model credentials or real secrets.
Build a local read-only MCP glossary server with the official Python SDK. Test discovery, valid calls, rejected inputs and subprocess shutdown.
Expose one documentation service as a resource, search tool and reusable prompt. Run discovery and use locally, then check host support and access boundaries.
Follow a documentation review to separate reusable instructions, MCP connectivity and executable tools, with examples that do not need all three.
Compare a direct HTTP API with an MCP adapter through the same local weather lookup, including discovery, schemas, access boundaries and maintenance costs.
Build a focused setup-guide review skill, validate its files, and design positive, negative, and failure-feedback tests without granting extra permissions.
Understand SKILL.md packages, staged loading, prompts, tools, and permission boundaries through a small review skill and runnable Python helper.
Build a local read-only order dispatcher to learn tool requests, argument validation, user permissions, controlled errors, and timeouts.
Compare chatbots, workflows, and AI agents through the same support ticket, with control flow, costs, failure cases, and a practical decision guide.
Follow an AI agent through support-ticket triage, run a bounded Python simulation, and learn when a script or workflow is the better choice.
Learn what MCP (Model Context Protocol) is, why it’s useful, and how to add MCP servers (Notion, Sentry, Context7) in OpenAI Codex, Claude Code, and Gemini CLI.
Learn how to secure your MCP (Model Context Protocol) server using FastAPI. Implement authentication, rate limiting, and input validation to protect your AI workflows from misuse, abuse, or injection.