Flow Basics: Natural Language Processing

The Flow Natural Language Processing (NLP) engine enables your bot to understand natural language and make informed decisions. The two tasks the NLP can perform are Intent recognition and Entity extraction. Intent recognition maps a user's utterance to a predefined bot reply. Entity extraction extracts specific data from a user's utterance. For these tasks to be successful, you must add training data to your flow.

Intent classification

The core of well-functioning conversational Artificial Intelligence (AI) is intent classification. Intent classification is the process of understanding user requests.

For example, the user utterance "I want to fly from Amsterdam to San Francisco" can be mapped to the intent of booking a flight.

NLP takes care of intent classification but in order to function, it needs to be trained with examples provided by the Flow conversational developer.

Adding a Triggers: Text lets you train the intent. When you click the Train button you can add examples for this intent.

Adding more examples enables the classifier to make better predictions. For best results, add a least ten examples that are diverse in form, but consistent with the context. When you have a sufficient amount of examples, you can click Save. Clicking Save initiates the training process.

📘

Note: If you add fewer than five examples, only the exact matching examples are used.

Reusing intents

A great way to speed up the development process is reusing intents. This also improves the quality of intent classification.

Every intent you create automatically becomes part of your intent library. You can drag and drop any existing intent in other flows depending on the context.

📘

Tip: We recommend creating separate and unique intents instead of duplicate intents.

Ensure you do not create semantically similar intents. If two intents are too similar, the probability of misclassification rises. In that case, the best practice is to create a more general intent and then ask follow-up questions.

Reuse example

Start by creating an intent to understand greetings (hi, hello, hi there, hey, etc). After training this intent with examples, reuse it by dragging and dropping it from the library.

Now, when a user starts a conversation with a greeting, the system will reply Hi! How can I help you?. If the user greets again, the system knows that they have already greeted, so it can reply with Hello again.

Follow up intents

The NLP engine works best if you provide clear examples. In some cases you can also train more generic intents. For example, if a user writes the closest sentence without any context, it doesn't mean much. However if the user asks after the contextual sentence ,then it becomes relevant. The AI engine makes it possible to create and train these intents in context, and then reuse them in other places.

So, if you have a flow on finding parking spaces, training the follow-up intent 'the closest' makes sense. You can then easily reuse this intent in a flow on finding restaurants or toilets.

Confusion

When designing flows, it's a good practice to think about the different use cases your bot will serve. One of the biggest challenges in flow design is keeping your project from offering too many identical intents that will confuse the NLP.

This can be best explained with an example.

Example

Imagine a company providing support for customers having problems with phones and dish washers. Per the business requirements, it might mean each product would result in a different answer. A broken phone might be picked up, while a broken dish washer requires a repair man to make an appointment.

We tend to split these repair questions into different flows, introducing a number of intents for repair questions regarding phones and dishwashers.

The training data for the intents might look like:

Repair phone intent

Hi, I have a broken screen on my new phone, can you fix this?
My phone is broken, I want it repaired
My phone locks up and won't start anymore
...

Repair dishwasher intent

Hi, I have a broken door on my new dishwasher, can you fix this?
My dishwasher is broken, I want it repaired
The dishwasher locks up and won't start anymore
...

Can you tell the difference? The problem is this will confuse the NLP because the example sentences are to similar and would make classification unreliable.

Merge intents by subject

One way to solve this issue is by combining the two intents into a single repair intent:

Repair intent

Hi, I have a broken screen on my new phone, can you fix this?
My phone is broken, I want it repaired
The dishwasher locks up and won't start anymore

With a single intent you would mark the products as entity types. Next, you can use the extracted product within reply actions and conditions to differentiate the follow-up flows.

Explanation

The NLP engine interprets sentences differently than humans do.

To humans, "I have an issue with my mother-in-law" has a totally different meaning than "I might have some problem with the television." To the system, both sentences are about someone having a problem with a certain object.

Since we cannot prevent this from happening, it's best to create intents based on subjects.

Read more

Entities instead of multiple intents

Other best practices

Balance the number of examples across intents
Try not to create multiple intents with just 10 examples and a number of intents with over 100 example sentences. It's better to keep intents balanced with around an equal amount of training examples.

The infamous Hello intent
When there is an intent with many short examples (often a 'hello' or greeting intent), short user utterances that are unclear are often misclassified as belonging to this 'hello' intent. Often it can be solved by turning off NLP for this intent.

Limit the number of entry intents
More intents results in a lower degree of classification accuracy, because the probability of choosing the wrong intent increases. Consider keeping the number of intents that can be matched at the entry level of a flow to a bare minimum. If your use case requires hundreds of entry intents, try to merge as many intents into the the same topic.

Keep your intents clean.
On the other hand, make sure you don't merge intents that are not semantically similar. Keep your intents clean and clear.