📺Monitor
Overview
The Monitor feature in ZBrain delivers continuous, end-to-end oversight of every deployed agent and application. It automatically captures each input and output, evaluates the output against a comprehensive set of metrics, and surfaces real-time performance trends, so you can detect issues early, correct them quickly, and maintain consistently high-quality AI interactions.
Monitor schedule evaluations at defined intervals, log both successes and failures, and present the results in an intuitive console. Built-in notifications keep you updated on whether the flow is successful or fails.
Metric categories
LLM-based -Utilizes a language model to evaluate answers for relevance and factual accuracy.
Response relevancy - Measures how accurately the response answers the user’s query.
Faithfulness - Evaluates factual alignment with context to minimize hallucinations.

Non-LLM - Relies on deterministic checks (health, exact match, similarity) without invoking an LLM.
Health check - Confirms the app/agent can return a valid response; halts further checks if an invalid response is received.
Exact match- Compares the app/agent agent response character-by-character with the expected output.
F1 score: Balances precision and recall to evaluate content overlap.
Levenshtein similarity: Measures similarity based on edit distance between two strings.
ROUGE-L score: Detects the longest common sequence between the response and reference text.

LLM-as-a-judge - Have an LLM emulate human reviewers on traits like creativity, clarity, and helpfulness.
Creativity: Rates originality in response generation.
Helpfulness: Evaluates how effectively the response assists in resolving the user’s query.
Clarity: Measures how clearly the message is conveyed.

Performance - Measures the total time (in milliseconds) taken by the LLM to return a response after receiving a query. It provides the Response Latency metric. Using this metric, users can monitor and enforce execution time thresholds for AI Agents, Flows, or Apps.
Set thresholds in either seconds or minutes.
When a threshold is breached or satisfaction is achieved, the system triggers Success or Failure evaluations and sends relevant notifications.


Key capabilities of Monitor
Automated evaluation: Assess responses using LLM-based and non-LLM-based metrics.
Performance tracking: Track success/failure trends.
Query-level monitoring: Configure evaluations at the individual query level within a session.
Agent and app support: Monitor both AI apps and AI agents.
Input flexibility: Monitor responses for .txt, PDF, image, and other file types.
Notification alerts: Enable real-time notifications for event status updates when an event succeeds or fails.
Monitor interface navigation
The monitor module consists of four main sections, accessible from the left navigation panel:
Events: View and manage all configured monitoring events
Monitor logs: Review detailed execution results and metrics
Event settings: Configure evaluation metrics and parameters
User management: Configure role-based user permissions


Together, these capabilities provide a single dashboard for validating fixes, identifying quality drift, and ensuring that every user interaction meets your organization’s standards.
Accessing Monitor documentation
ZBrain Builder includes a context-aware documentation panel within the Monitor module.
Click the '?' icon in the top-right corner of the Monitor interface.

When clicked, it opens a side panel that displays documentation specifically tailored to the Monitor module, all without requiring you to navigate away from your current screen.

Last updated