How to Build an Intelligent Customer Service (Part 3): Entity Extraction and Application in NLP

[baoshi.rao]

The previous two articles discussed user intent segmentation and recognition, and the multi-turn dialogue flow designed based on it. This article continues to explore the extraction of entity information in natural language processing (NLP) and its application in products, as the third installment in the series on building an intelligent customer service system.

You might have wondered, how does AI understand human speech? For example, when I tell a robot in a shopping mall that I want to go to the third floor, how does it understand my meaning? Or when I say 'go to NARS,' how does it know what NARS is and where it is located?

To answer these questions, we need to talk about entities.

First, what does entity information mean?

It refers to key pieces of information, which can generally be divided into two types: 1) information largely unrelated to the business, which can be considered common knowledge, such as phone numbers, email addresses, dates, and times; and 2) business-related information, customized according to specific scenarios.

How to understand this? Let me illustrate with examples.

Example 1: Booking a Flight

Suppose you want to book a flight ticket and say:

'Help me book a ticket for tomorrow from Beijing to Shanghai, departing in the afternoon. My phone number is 13344445555, and send the itinerary to my email: yuanquaner@woshipm.com.'

'Okay, I’ve found the following flights that meet your criteria...'

In this scenario, 'tomorrow' and 'afternoon' are time-related information, 'Beijing' and 'Shanghai' are location information, and '13344445555' is a phone number. Since these are common knowledge and not tied to a specific business, they are typically built into the algorithm for recognition and conversion.

Here, the AI first identifies the date and time, converts 'tomorrow' to 'March 27, 2020,' and 'afternoon' to '14:00–18:00,' then combines this with other information to request data from the airline and return the results—a list of eligible flights.

This recognition step sounds simple because humans reflexively understand what a date, phone number, or location is. But for AI, these are initially just strings of characters that require extensive algorithms to distinguish and extract. The recognition rate here is closely tied to the richness of the algorithm's language understanding.

For example, if the algorithm initially recognizes 'tomorrow' and 'yesterday' but overlooks 'the day before' or 'the day after,' or recognizes 'February 2, 2020' but not 'the second of February, 2020,' the recognition rate drops—leading to what we might call 'higher stupidity rates.'

An interesting point about conversion:

Human expressions of dates and times in natural language often don’t follow strict logic. For instance, if you tell a smart assistant at 00:10 AM, 'Set an alarm for tomorrow,' 'tomorrow' likely means 'today.' Or if you say in January 2020, 'Check my December bill,' you’re referring to December 2019. These linguistic habits must be accounted for to make the assistant 'feel human.'

Siri handles this well. In the alarm scenario, it might confirm, 'Did you mean to set an alarm for 8 AM today?'

Example 2: Business-Specific Entities

Now, let’s look at a business-specific scenario. Suppose you want to order bubble tea via a smart assistant:

'I’d like a bubble tea from Yi Dian Dian, a large Four Seasons Milk Tea, full sugar, with red beans, no ice.'

'Okay, order placed.'

Here, 'Yi Dian Dian' is the brand, 'Four Seasons Milk Tea' is the product name, 'large' is the size, and 'full sugar' is the sweetness level. These terms are entirely business-dependent and lose meaning outside this context. For example, saying 'Yi Dian Dian' in a weather conversation would confuse the listener. Unlike 'Beijing,' which is always 'Beijing,' these terms are scenario-specific.

Thus, such information must be customized for the actual scenario. In other words, the AI must be taught what a 'bubble tea brand,' 'product name,' or 'size' is.

How to teach the AI?

There are three main methods: string matching, regular expressions, and annotation.

1. String Matching: Define a set of entity values for fixed terms. For example, in the bubble tea scenario:

   'Bubble Tea Brand' = ['Yi Dian Dian', 'Hey Tea', 'Lele Tea', 'Nayuki Tea']

   'Size' = ['Large', 'Medium', 'Small']

   'Product Name' = ['Four Seasons Milk Tea', 'Black Tea Macchiato', 'Cheese Ballet']

   When a user mentions one of these, the AI knows what they’re referring to.

2. Regular Expressions: Define rules for patterned information. For example, a license plate number might follow: province abbreviation (one Chinese character) + one letter + five digits. In pseudocode:

   'License Plate' = [京沪鲁浙…津豫赣]{1}[A-Z]{1}[0-9]{5}

   This method works for other patterned terms like device batch numbers, ID numbers, phone numbers, or order numbers.

3. Annotation: Label expressions of a concept to train the AI, suitable for irregular or non-fixed expressions. For example, in a loan repayment scenario, users might ask:

   'How much have I repaid?'
   'How much loan have I paid off?'
   'What’s the remaining amount I owe?'

   These lack fixed terms or clear patterns but share semantic meaning. By annotating phrases like '[repaid amount]' as 'paid' and '[remaining amount]' as 'remaining,' the AI learns to recognize these entities.

After extensive annotation, the AI can identify such entities and continue the conversation 'like a human.'

In practical implementation scenarios, the methods of extraction are not unique or necessarily fixed. The specific approach adopted to meet requirements is determined by the PM's understanding of technical methods and familiarity with the business context. Ultimately, everything is done to serve user needs.

That's all for today's content~ The next topic will be on how to make AI seem more human—simulating a sense of intelligence. Thanks for watching!