How to configure a monitoring event for apps
Step 1: Access the monitoring configuration
To set up monitoring for an application:
Access the app session:
Navigate to the Apps page
Click on your desired application
Go to the query history section of the app
Select a specific user session from the list to monitor (shows session ID, user info, prompt count)
Review the conversation:
View the session details and chat history. You can configure monitoring events at the query level. For instance, in a conversation containing multiple queries, each query can have its own monitoring event
Examine the response options (copy, conversation logs, edit annotation reply and feedback)
Access conversation logs:
Click 'Conversation Log' to see the interaction details of a specific query
Review status, time, and token usage
Check the input, output, and metadata
Enable monitoring:
Click the ‘Monitor’ button in the overview tab
Click ‘Configure now’ when prompted with ‘Added for monitoring’
Step 2: Configure event settings
You will be redirected to the Events > Monitor page. In the last status column, click ‘Configure’ to open the event settings page. On the event settings screen:
Review entity information
Entity name: The name of your application
Entity type: The type of entity being monitored (e.g., App)
Verify monitored content
Monitored input: The query or prompt being evaluated
Monitored output: The response being assessed
Set evaluation frequency
Click the dropdown menu under "Frequency of evaluation."
Select the desired interval (Hourly, Every 30 minutes, Every 6 hours, Daily, Weekly, or Monthly)
Configure evaluation conditions
Click ‘Add metric’ in the Evaluation Conditions section
Select a metric type from the available metric types:
LLM-Based
Non-LLM Metrics
LLM-As-Judge
Choose the evaluation method (is less than, is greater than, or equals to)
Set the threshold value (0.1 to 5.0)
Click ‘Add’ to save the metric
Set the "Mark evaluation as" dropdown to fail or success
Configure notifications
Toggle the ‘Send Notification’ option to enable alerting for this monitoring event.
Click ‘+ Add a Flow from the Flow Library’. This opens the Add a Flow panel.
In the panel, search for the desired notification flow and select it.
You can configure multiple notification flows for a single monitoring event, allowing you to send alerts simultaneously across various channels.
The system prevents adding the same flow more than once to avoid redundant alerts or conflicts.
Upon meeting the evaluation trigger conditions, all attached flows are executed in parallel.
You can directly edit linked notification flows from the Event Settings screen, allowing you to quickly update or troubleshoot notification logic without needing to navigate away from the monitoring configuration as it directly redirects you to the Flow page.
Click on the pencil icon
beside the selected Flow.
It will redirect you to the defined Flow page, and here you can modify the logic.
Click the Play ▶️ button to run a delivery test.
If the flow succeeds, a confirmation message appears: "Flow Succeeded".
If the flow fails, inline error messages will be displayed, along with a link to Edit Flow for troubleshooting.
Note: Users cannot update the event settings until a valid notification flow passes the delivery test. Once the flow passes the delivery test, notifications will be sent via the chosen communication channel whenever the associated event fails.
Test your configuration
Click the ‘Test’ button
Enter a test message if needed
Review the test results
Click ‘Reset’ if you want to try again
Save your configuration
Click ‘Update’ to save and activate your monitoring event
Last updated