How to configure a monitoring event for agents

Step 1: Access the monitoring setup

To set up monitoring for an agent:

Access the agent dashboard:

• Go to the Agents page.

• Choose a deployed agent. This opens that specific agent’s dashboard.

Enable monitoring:

• Click on the full-screen button of ‘Agent Activity.’

• Click the 'Monitor' button of that execution.

• When prompted, select configure ‘Now.’

Step 2: Configure event settings

When you click on configure ‘Now,’ you’ll be redirected to the ‘Monitor’ page. Click ‘Configure’ in the ‘Last Status’ column to open the ‘Event Monitoring Settings’ page.

On the Event Monitoring screen, review the following settings, similar to the app monitoring configuration:

• Entity Name / Type (Agent)

• Monitored Input (e.g., a prompt or document)

• Monitored Output (response generated by agent)

Configure the following options:

• Evaluation Frequency (Hourly, Daily, etc.)

• Metrics

ZBrain enables you to monitor AI agent performance through customizable evaluation conditions, utilizing a comprehensive set of metrics. These metrics are configurable under the Evaluation Conditions section of an event.

• Adding Metrics

  • Click the “Add Metric” button under the Evaluation Conditions section.

  • Choose from three metric categories:

    • LLM-Based

    • Non-LLM Metrics

    • LLM-As-Judge

  • Combine multiple metrics using AND / OR logic to define composite evaluation rules.

Note: ZBrain continually expands its list of available metrics to support advanced evaluation needs.

• Setting thresholds

  • After choosing metrics, configure thresholds based on your business requirements.

  • Start with a conservative range (e.g., 0.5–0.7) and fine-tune based on performance.

• Consider the context:

  • Customer-facing applications → higher thresholds

  • Internal tools → lower thresholds are acceptable

  • Revisit and adjust thresholds as models or use cases evolve.

• Define outcome and notifications

  • Set the outcome of evaluation as either Success or Fail from the Mark Evaluation drop down, depending on the metric logic.

• Enable notifications by toggling the Send Notification button if you want to get notifications on agent events.

Notifications are triggered only when conditions are met.

• Adding notification flows

  • Click '+ Add a Flow from the Flow Library'

  • In the side panel, search and select your preferred notification flow. You can add multiple flows for a single event, allowing you to send alerts simultaneously across various channels. The system prevents adding the same flow more than once to avoid redundant alerts or conflicts. Upon meeting the evaluation trigger conditions, all attached flows are executed in parallel.

• You can directly edit linked notification flows from the Event Settings screen, allowing you to quickly update or troubleshoot notification logic without needing to navigate away from the monitoring configuration, as you are directly redirected to the Flow page.

  • Click on the pencil icon beside the selected Flow.

  • It will redirect you to the defined Flow page, and here you can modify the logic.

• Click the Play ▶️ button to run a delivery test.

• If the flow succeeds, a confirmation message appears: "Flow Succeeded".

• If the flow fails, inline error messages will be displayed, along with a link to Edit Flow for troubleshooting.

Note: Users cannot update the event settings until a valid notification flow passes the delivery test. Once the flow passes the delivery test, notifications will be sent via the chosen communication channel whenever the associated event fails.

• Test the event

  • Click the ‘Test’ button to simulate a monitoring event.

  • Optionally, enter a test message to preview the response evaluation.

  • Review the test results for accuracy.

  • Click ‘Reset’ to test again with different inputs.

• Save and activate

  • Once all settings are configured:

    • Click ‘Update’ to save your Monitor configuration.

    • The Monitor runs automatically at the selected evaluation frequency:

      • Hourly

      • Every 30 minutes

      • Daily

      • Weekly