📺Monitor
Overview
The Monitor feature in ZBrain offers comprehensive oversight of your AI agents and applications by automating evaluation and performance tracking. It ensures response quality, helps identify issues proactively, and maintains optimal performance across all deployed solutions.
Monitor captures inputs and outputs from your applications, continuously evaluating responses against defined metrics at scheduled intervals. This provides real-time insights into performance, tracking success and failure rates, and highlighting patterns needing attention. The results are displayed in an intuitive interface, enabling you to quickly identification and resolution of issues and ensure consistent, high-quality AI interactions.
Key capabilities
Automated evaluation: Assess responses using LLM-based and non-LLM-based metrics.
Performance tracking: Track success/failure trends.
Query-level monitoring: Configure evaluations at the individual query level within a session.
Agent and app support: Monitor both AI apps and AI agents.
Input flexibility: Monitor responses for .txt, PDF, image, and other file types.
Monitor interface navigation
The monitor module consists of three main sections, accessible from the left navigation panel:
Events: View and manage all configured monitoring events
Monitor logs: Review detailed execution results and metrics
Event settings: Configure evaluation metrics and parameters
How to configure a monitoring event for apps
Step 1: Access the monitoring configuration
To set up monitoring for an application:
Access the app session:
Navigate to the Apps page
Click on your desired application
Go to the query history section of the app
Select a specific user session from the list to monitor (shows session ID, user info, prompt count)
Review the conversation:
View the session details and chat history. You can configure monitoring events at the query level. For instance, in a conversation containing multiple queries, each query can have its own monitoring event
Examine the response options (copy, conversation logs, edit annotation reply and feedback)
Access conversation logs:
Click 'Conversation Log' to see the interaction details of a specific query
Review status, time, and token usage
Check the input, output, and metadata
Enable monitoring:
Click the ‘Monitor’ button in the overview tab
Click ‘Configure now’ when prompted with ‘Added for monitoring’
Step 2: Configure event settings
You will be redirected to the Events > Monitor page. In the last status column, click ‘Configure’ to open the event settings page. On the event settings screen:
Review entity information
Entity name: The name of your application
Entity type: The type of entity being monitored (e.g., App)
Verify monitored content
Monitored input: The query or prompt being evaluated
Monitored output: The response being assessed
Set evaluation frequency
Click the dropdown menu under "Frequency of evaluation"
Select the desired interval (Hourly, Every 30 minutes, Every 6 hours, Daily, Weekly, or Monthly)
Configure evaluation conditions
Click ‘Add metric’ in the Evaluation Conditions section
Select a metric type:
LLM-based metrics
These metrics assess the quality of responses generated by large language models:
Response relevancy: Checks how well the response answers the user's question. Higher scores mean better alignment with the query.
Faithfulness: Measures whether the response accurately reflects the given context, minimizing hallucinated or incorrect information.
Non LLM-based metrics These metrics rely on algorithmic comparisons to evaluate responses without involving language models:
Health check: Determines whether an entity is operational and capable of producing a valid response. Note: When the health check fails, no further metric evaluations are performed for that execution.
Exact match: Compares the response to the expected output for an exact character-by-character match.
F1 score: Balances precision and recall to assess how well the response captures the expected content.
Levenshtein similarity: Calculates how closely two strings match based on the number of edits needed to transform one into the other.
ROUGE-L score: Evaluates similarity by identifying the longest common sequence of words between the generated and reference responses.
LLM as a judge metrics:
These metrics simulate human-like evaluation by using an LLM to judge qualitative aspects of a response:
Creativity: Rates how original and imaginative the response is in addressing the prompt.
Helpfulness: Assesses how well the response guides or supports the user in resolving their query.
Clarity: Measures how easy the response is to understand and how clearly it communicates the intended message.
Choose the evaluation method (is less than, is greater than, or equals to)
Set the threshold value (0.1 to 5.0)
Click ‘Add’ to save the metric
Set the "Mark evaluation as" dropdown to fail or success
Configure notifications
Toggle the ‘Send Notification’ option to enable alerting for this monitoring event.
Click ‘+ Add a Flow from the Flow Library’. This opens the Add a Flow panel.
In the panel, search for the desired notification flow and select it.
Click the Play ▶️ button to run a delivery test.
If the flow succeeds, a confirmation message appears: "Flow Succeeded".
If the flow fails, inline error messages will be displayed, along with a link to Edit Flow for troubleshooting.
Note: Users cannot update the event settings until a valid notification flow passes the delivery test. Once the flow passes the delivery test, notifications will be sent via the chosen communication channel whenever the associated event fails.
Test your configuration
Click the ‘Test’ button
Enter a test message if needed
Review the test results
Click ‘Reset’ if you want to try again
Save your configuration
Click ‘Update’ to save and activate your monitoring event
How to configure a monitoring event for agents
Step 1: Access the monitoring setup
To set up monitoring for an agent:
Access the agent dashboard:
Go to the Agents page.
Choose a deployed agent. This opens that specific agent’s dashboard.
Enable monitoring:
Click on the full-screen button of ‘Agent Activity.’
Click the 'Monitor' button of that execution.
When prompted, select configure ‘Now.’
Step 2: Configure event settings
When you click on configure ‘Now,’ you’ll be redirected to the ‘Monitor’ page. Click ‘Configure’ in the ‘Last Status’ column to open the ‘Event Monitoring Settings’ page.
On the Event Monitoring screen, review the following settings, similar to the app monitoring configuration:
Entity Name / Type (Agent)
Monitored Input (e.g., a prompt or document)
Monitored Output (response generated by agent)
Configure the following options:
Evaluation Frequency (Hourly, Daily, etc.)
Metrics: Use LLM or non-LLM-based scoring
Choose the evaluation method and set the similarity threshold.
Outcome (mark as Success or Fail)
Click ‘Test’ and then ‘Update’ to save.
Once activated, the monitoring event runs automatically at the specified evaluation frequency, such as every 30 minutes, daily, or weekly.
Events dashboard
The events dashboard displays all configured monitoring events in a tabular format:
Entity name: The agent or application being monitored
Entity type: Classification (App, Agent, Playground, etc.)
Input: The query/input being evaluated
Output: The response being assessed
Run frequency: How often does evaluation occur
Last run: When the last evaluation occurred
Last status: Current status with the ‘Configure’ option
Use the search box and dropdown filters, Entity (App/Agent/Playground) and Status (All, Success, Failed), to quickly locate specific events. Click ‘Configure’ in the last status column to modify event settings.
Monitor logs
The monitor logs interface provides detailed performance tracking:
Key components
Event information header
Event ID: Unique identifier for the monitoring event
Entity name: The agent being monitored
Entity type: Classification (App, Playground, etc.)
Frequency: How often monitoring occurs
Metric: Performance criteria being measured
Log status visualization
Colored bars provide a quick visual indicator of recent execution results
Red bars indicate failures, green indicates successful evaluations
Filtering options
Status dropdown: Filter by All/Success/Failed/Error status
Log time dropdown: Filter by active/inactive
Log details table
Log ID: Unique identifier for each log entry
Log time: When the evaluation occurred
LLM response: The query or prompt content
Credits: Resource utilization
Cost: Associated expense
Metrics: Success (✅), Failure (❌)
Status: Outcome (Success/Failed/Error with color coding)
Setting appropriate thresholds
Start with conservative thresholds (0.5-0.7) and adjust based on observed performance
Consider your use case requirements when setting thresholds:
Customer-facing applications may require higher thresholds
Internal tools might tolerate lower thresholds
Regularly review and adjust thresholds as your applications evolve
Troubleshooting
Investigate failed evaluations by reviewing the specific LLM responses
Check metric scores to understand why responses did not meet thresholds
Adjust prompts or application configuration based on monitoring insights
User management in monitor
ZBrain Monitor supports role-based user permissions. Admins can assign access to specific users who can view and configure monitoring events.
How to manage users
From the Monitor page, select any monitoring event.
Navigate to the ‘User Management’ tab.
A user management panel opens with the following elements:
Entity name: Name of the agent/app.
Entity type: AGENT or APP
Builder: Select the builder you want to invite. The options include custom or everyone.
When selecting the ‘Custom’ option, you can use the search field to locate builders to invite. Enter the user’s name and click the ‘Invite’ button to assign them the Builder role.
Once accepted, the user will see the Monitor tab upon login and will be able to manage the assigned event.
Last updated