1 - Introduction
As a xBot manager, you may want to know:
-
If your xBot is performing accurately?
-
What your xBot is being used for and who is using it?
-
See conversations/transcripts of dialogue between your customers and your xBot?
The following reporting pages will help you to optimize your xBot. You will access a lot of valuable information such as:
-
key indicators on past and real-time data
-
anonymized data giving a better understanding of what your customers want
-
recommendations on which Intents should be optimized
2 - Reporting Interface
You can access the Reporting page by clicking on the icon on the vertical menu on the left side of the my.satisfaction.AI portal.
All reporting pages use the same framework. You can select on the left which reporting you wish to see and in the navigation bar, you can adjust the display of your reporting via 3 criteria:
-
the xBot environment: your xBot in Production and the xBot in Development (i.e pre-production)
-
the timezone: timezone when/where the conversations occurred
-
the timeframe: pre-selected or custom period when the conversations occurred
- Moreover, all graphics can be exported (via a dedicated menu on the top right) as PNG images and the associated data can be exported as CSV files:
For more information on how to import a CSV file in Microsoft Excel, you can consult this documentation:
. http://www.wikihow.com/Open-CSV-Files
- NB:
-
All reports are updated every quarter of an hour (= at xx:00, xx:15, xx:30, xx:45)
-
The format of the calendar is an American format (week starts on Sunday). This will evolve in a future release.
-
3 - Live Section
A - Right now page
In the Live section of the Reporting page, you can find via the Right now indicator the number of active sessions that are running for the last 20 minutes.
By sessions, we mean dialogues between your users and the xBot with at least 1 user input.
4 - User management section
A - Users page
In the User management section of the Reporting page, you can see 2 graphics in the Users page illustrating:
-
the evolution of the Total active users : Total number of users who have received AND sent at least a message
-
the evolution of the Total users : Total number of users who have received OR sent at least a message
5 - Conversation section
A - Messages page
In the Conversation section of the Reporting page, you can see 3 groups of graphics in the Messages page illustrating:
-
the number of messages received / sent
-
the average number of messages sent and received per session
-
the average duration of session time
B - Transcripts page
In the Conversation section of the Reporting page, you can see in the Transcripts page the list of the verbatim, of what has been said between the users and your xBot:
If you want to perform custom analysis on Microsoft Excel or OpenOffice Calc spreadsheets, you can export this list of transcripts in a CSV file.
UTF-8 is the character set to use to have the accents and special characters correctly displayed. For more information on how to import a CSV file in an Excel spreadsheet, please consult this documentation .
The transcripts list can be sorted by several criteria such as:
-
the Duration of the conversation
-
The number of users’ inputs Received by the xBot
-
The number of user’s inputs Sent by the xBot
-
The number of Steps or exchanges that have taken place. These steps are calculated = [(user’s input + xBot’s answer) / 2]
-
The Sentiment analysed in the conversation
-
If the conversation has been Waiting to be handled to a human agent or not
-
If the conversation has been Handled by a human agent or not
For more information on the three last criteria, please consult the documentation related to the Sentiment analysis.
On the right side, via the “play” 
button you can also access the transcript, the “word for word” records of what was said during a conversation:
1 - Transcripts logs and information
The transcripts contain a lot of useful information for each sentence that will allow you to improve the relevancy of the xBot’s future answers.
For each transcript, you can find the following information on the top of the page:
-
The number of Intents triggered
-
The list of the Intents triggered. If the Intent is elected several times, it will be shown here only once. The Default Intent (if triggered) will not be displayed here.
- Within the transcripts is present different type of information:
a) - Sentiment Analysis of the sentence
-
For each transcript a Sentiment Analysis* of the sentences is performed. You also have the possibility to provide crowdsourced feedback on:
- the sentiment that has been associated to the end user inputs
&
- the relevancy of the xBot’s response
- Sentiment Analysis:
The sentiment analysis is performed on the information exchanged between the end user and the xBot. This analysis is based on our Dialog AtmosphereTM rating system.
You can find more information on the sentiment analysis and how this rating system works on this page.
b) - Information on the elected Intents
Every sentence addressed to your xBot is processed to be matched with an intent. Below each User sentence recognised by the AI model of your xBot is displayed the name of the elected Intent, or in other words the Intent set in the Knowledge & Skills page and also in the Intents page of the portal.
In the transcripts four different ways of understanding Intents can be displayed:
Recognition of user intent by the xBot can have four results:
1) Perfect match: Your AI model matches the sentence with one of your intent:
= Match to a known Intent
Example:
=
= Match to a clicked button
Example:
Note: The text “yes” displayed above is the one in this example set here ]yes[ in the Athena code below:
? BUTTON CHOICE
[list][*][button title="Yes" ]yes[/button][*][button title=”no”]no[/button][/list]
2) 
= **Match to “data extract” or “off-topic“
This corresponds to:
- Data Extract: This is displayed in the transcripts for all scenarios that need a specific response in a certain format to go further in the conversation.
Example:
- or to an “Off topic” in the user’s response to the conversation.
Example:
The handling of “off-topics” in a dialog scenario can be defined in the xBot knowledge with either an Athena © Pattern such as the Default (by using the Athena © syntax “?”) or with an Athena © Pattern such as the Default forced (by using the Athena © syntax “???”.
This tagging of “off-topic” sentences may be useful in sub-scenarios, in order to capture all responses provided by your end users that do not match the response expected by the xBot in a particular scenario. For example, for the question”What is your name?” the expected answer is a “Name”. All other answers are answered by the Athena © Pattern Default “Sorry, I didn’t understand that”.
- Possible Optimization: All sentences in transcripts that are marked with a Data Extract or Off topic label with this icon are sentences that you can now analyze to add new response scenarios to your xBot or to adjust existing scenarios.
3) 
= Need for disambiguation because the user’s sentence does not correspond to only one known Intention by the xBot
- The matching score between two intents is so close that the xBot needs a disambiguation (clarification) from the user. For more information on what disambiguation is, please visit this page.
Example:
- Possible Optimization: When you detect that the user regularly expresses several intents within the same sentence in a scenario, you can rework the conversational tree of your scenario to help your xBot to respond better.
4) 
= [DEFAULT] Label triggered because the user’s sentence has not been understood
- No intent can be associated because your xBot does not yet have the knowledge or skill requested by the user. Sentences detected in that way are displayed with an icon below.
Example:
- Possible Optimization: All you have to do here is to give to your xBot this knowledge in the Knowledge & Skills page of the portal, then feed your xBot in classified sentences and finally train your xBot so that it can answer these questions.
2 - Events and advanced informations
By clicking on the “Show all events” button, you can access additional technical information:
A click on the End user sentence unfolds additional custom data and User/xBot events information such as:
-
custom data & message: information logged here depends on the specifications of the external connector (Facebook Messenger, Slack, Microsoft Teams, Skype, etc…) used to host the conversation channel and how integration has been set
-
xbrain-dialog-atmosphere: score, polarity, offensive, emojis, dialogScore, etc … information
-
Athena log information: Classifier and debug information
3 - Transcripts Export
You can export all the content of the transcripts for your own custom analysis. The output file is a CSV file.
UTF-8 is the character set to use to have the accents and special characters correctly displayed. For more information on how to import a CSV file in Microsoft Excel, please consult this documentation .
Bot management section
A - Intents page
In the Intents page you can see the Intents activity during the selected period :
-
On the vertical axis is displayed the Intents sorted by the most triggered/detected intents
-
On the horizontal axis the number of the times the Intent is triggered is displayed.
B - Bot optimization page
In the Bot optimization page you can see the number of sentences classified per intents:
-
On the vertical axis is displayed the Intents sorted by the most popular tasks or actions the user ask for
-
On the horizontal axis the number of sentences classified per Intents is displayed.
Recommendations:
For effective and balanced xBot training, you should classify at least 5 sentences per intent. We recommend also a maximum of 30 sentences per intent.
At the bottom of this report, you can find illustrated which Intents are unbalanced and should be rebalanced before the next training session.
Documented/reviewed 03/04/2018 17:12:55