5 New Skills Product Managers Need to Master in the AI Era
-
Unlike traditional product managers, those in the AI era focus more on applying technology to business problems. The most critical responsibility of an AI-era product manager is to provide data specifications, which requires a deep understanding of data. This article summarizes the new skills product managers need to master in the AI era for collective learning and reference.
On January 25, 2020, at the Manchester Product Thinking Conference, Mayukh Bhaowal, Product Director of Salesforce's Einstein system, shared insights on the adjustments product managers need to make in the AI era and how to build successful AI products. The presentation was titled How AI Is Changing The Product Management Job Description by Mayukh Bhaowal.
Before diving into the main topic, let's start with some interesting stories:
A few months ago in Cambridge, Massachusetts, hundreds of tech enthusiasts gathered as a Stanford professor wrote on a whiteboard. He discussed how AI is transforming the world and how traditional professions are being disrupted and dismantled by AI. This professor was Andrew Ng, founder of Landing AI and co-founder of the renowned online education platform Coursera. During his talk, he described the evolving role of product managers.
Take a chatbot application as an example. In the internet era, if a product manager was designing a new webpage or app, wireframing was an essential part of the standard process. Through visuals, product managers could communicate to engineers what the program should look like, and engineers would implement it accordingly. This has long been the collaborative model between product managers and engineers in Silicon Valley and tech companies.
But in the AI era, this old way of working is being dismantled.
What if you want to build a chatbot? I once helped a company develop a chatbot called Behavioral Therapy, designed to address mental health issues among Americans through conversational interactions.
If depicted in a wireframe, it might look like this:
- Bot: "Hi~"
- User: "I'm unhappy."
- Bot: Displays a magical image (generated via natural language processing).
I often say, "This is completely useless." I don’t need to know the shape of the speech bubbles; I need to understand the substance of the conversation—how my chatbot can perceive what the user is experiencing.
Traditional product managers provide specifications for chatbots through wireframes, but engineers need to understand the underlying logic.
In San Francisco, California, there’s an organization called Insight Data Science.
Recently, they launched a seven-week intensive program to help professionals from diverse backgrounds transition into software engineering and data science. My French colleague, Léon McGuire, who holds a Ph.D. in neuroscience, joined their data science program and secured a job as a data scientist at Lincoln.
Recently, I learned from Jeremy Carrasco, an AI product lead, that they are launching a brand-new program called the Data Product Manager or AI Product Manager program. Jeremy believes that traditional product managers need to acquire essential skills to focus on data and leverage AI to build products.
Finally, Marco Casalaina, VP of Product at Salesforce Einstein, is collaborating with his team to develop a new internal training course to prepare for the fourth industrial revolution driven by AI and machine learning.
I asked Marco, "What’s the goal of this course?" He replied, "We want our product managers to develop an intuition for quickly assessing the feasibility of using machine learning to solve business problems—to know which problems are suitable for ML or AI and which are not."
How is product management changing in the AI era? As a product manager, what skills do you need to master to build successful AI products?
Engineering as a discipline has existed for thousands of years, dating back to the construction of the Egyptian pyramids and military engines. Over a millennium, mechanical engineering emerged, steam engines were invented, and engineering management became a well-established field.
In contrast, product management is much younger, with a history of less than a century. The earliest product managers were essentially brand managers, referred to as brand men.
In manufacturing, the role evolved into product line managers. With the rise of software, it transformed again, adopting agile methodologies like Scrum.
This reminds me of professional sports. Over the past 50 to 100 years, TV commentary has replayed every breakthrough and change in sports. Whether you’re participating on the field or watching comfortably at home, this has fundamentally altered how you engage with sports and where you place your emotional investment.
The movie Moneyball popularized sports analytics. Additionally, high-tech equipment and gear appear at every Olympic Games, setting new world records every four years. We are indeed on a path of higher, faster, stronger. Finally, more women are entering professional sports. The Washington Post even highlighted the growing leadership of women in traditionally male-dominated sports.
Similarly, product management is like a sport—it’s being redefined by the advancements in AI and software. If you reflect on the stories I’ve shared, you’ll notice emerging metrics for product managers. The world is changing, and AI product managers are on the rise.
Traditionally, product managers bridge cross-functional stakeholders like sales, marketing, and development. But as an AI product manager, you must also connect two additional key roles: data scientists and data engineers.
At the same time, product managers need to update their skill sets in five areas:
- Problem Mapping
- Data Literacy
- Acceptance Criteria
- Explainability, Ethics, and Bias
- Moving Research to Production
With the AI boom, you may face new execution pressures to integrate AI into products. But we often overlook mapping these technical solutions back to business problems.
I believe AI product managers must articulate the product’s value proposition clearly. They should first consider traditional methods and evaluate rule-based engines before taking risks.
Let’s take an example of improving customer service efficiency.
Large companies like Amazon and Uber have massive customer service departments. Every day, countless new cases arise. For instance, customers might complain:
- "Where’s my order? Why hasn’t it arrived?"
- "I received the wrong order; I need a replacement."
- "I need a refund. The Uber driver canceled my ride, but the charge hasn’t been reversed!"
This department’s goal is to resolve these cases as quickly as possible. Thus, the system’s metric is minimizing case resolution time.
As a product manager, I’d first consider the problem, traditional methods, processes, and rule-based engines. The best engines have been in use for some time.
In this case, there’s an issue: the category and product fields are empty.
These fields help route cases to the correct department for swift resolution. Without them, cases may be misassigned, bouncing between departments and wasting valuable time.
Perhaps every product manager has thought this: by using certain rules, the values in this field can be filled based on other fields in customer service cases. However, upon further analysis, we find that such rules are cumbersome. They do not scale over time and become difficult to manage. Frankly, sometimes a rigid rule fails to truly capture the value.
But if you have a piece of free-form text lacking key fields and descriptions, you cannot derive rules to map it to 'product' and 'category.' This is where AI comes into play. In fact, this can be modeled as a multi-classification problem. Each value in these fields corresponds to multiple classifications. You can manually learn and derive insights from historical customer service cases—for example, by examining titles and descriptions to determine what the predicted value should be.
In our 'Einstein' system, we can see the predicted values for these fields. Each prediction comes with a corresponding confidence level. People can evaluate these using traditional manual methods rather than risking AI-based solutions. Remember, we must always map the solution back to the business problem—namely, reducing the resolution time for customer service cases.
What is the role of product managers in the AI era? I believe one of their most critical responsibilities is to provide data specifications. As we know, data is the foundation of any machine learning algorithm.
The first question we must ask is: Do we have enough data? If not, there is no dataset to train the machine learning model, and thus no way to learn from predictive signals in the dataset.
The second question is: How clean or noisy is your data in the real world? From what we’ve seen, most data is messy and noisy. It may reside in third-party systems. Before feeding it into machine learning, you may need to integrate data from various sources and import it into a data warehouse.
The third question is: Does historical data contain labeled examples for supervised machine learning? This is the so-called supervised classification problem. Without labeled examples to train the model, data scientists must find alternative approaches.
Take a general-purpose AI product for image recognition as an example:
General-purpose datasets can often be found online. If you use this to classify cats and dogs, its performance might astonish you—close to 100% accuracy. But if you apply the same product to detect tumors in medical diagnostics, its performance would be poor because it has never seen or been trained on such data. In reality, you might not even have labeled data to meet the training requirements for machine learning.
It must be emphasized that data introduces a completely new dimension, one absent from traditional product documentation. As some say about intelligent products: Data is the new user interface; data is the new user experience.
Before deploying a product in the real world, what are its acceptance criteria? In traditional product management, this might include feature completeness, page load times, or appropriate completion prompts. For intelligent products, you must also consider data science metrics like accuracy, precision, and recall.
Consider an example of fraud classification, where we attempt to separate fraudulent transactions from legitimate ones.
This is an example of an ideal classifier that correctly labels every fraudulent transaction as fraud and every legitimate one as benign. Data scientists would call this perfect precision and perfect recall, but this is an unattainable dream classifier.
In reality, it may lean in two directions: either missing some fraud cases while maintaining perfect precision (left figure below) or identifying fraud but also misclassifying legitimate cases as fraud, introducing false positives (right figure below).
As a product manager, it’s your responsibility to define the correct evaluation metrics for the domain, enabling data scientists to take appropriate action. In this case, the product manager doesn’t need the wisdom of a rocket scientist but must understand that fraudulent transactions are harmful and their cost far outweighs the introduction of false positives.
But this isn’t enough. Once you’ve identified the right metrics, you must also determine the threshold—when should the model achieve what you consider a reasonable value? Should precision be 80% or 90%? When deciding this, you must again consider business metrics and trace them back to data science metrics.
Take another example from sales: lead scoring.
We’re trying to predict the likelihood of a lead converting into a sale. For instance, Mr. Greg Thomson has a score of 88, meaning an 88% chance of converting into an order. The tricky part is that this is a conversion rate, and we must also consider the overall conversion funnel.
Two key metrics emerge here: one is the business metric determined by the product manager based on business needs; the other is the data science metric, derived in collaboration with data scientists and engineers. The latter helps build competitive barriers in business.
Thus, when preparing to launch an AI product to real users, you can first define these additional acceptance criteria.
Today, many of our customers are experiencing our predictive applications. The most common question is: 'Why didn’t the machine learning model make the decision it did?'
In reality, AI software is fundamentally different from traditional software because its outcomes aren’t generated from a set of pre-written code. Over time, as data and feedback loops deepen, the software becomes more like a black box. But we must also consider explainability, as it helps build user trust in the product.
Here’s a diagram to help evaluate explainability versus accuracy:
The graph shows a significant trade-off between explainability and accuracy in machine learning. Simpler models like linear regression or decision trees offer lower accuracy but better explainability, while neural networks and deep learning provide high accuracy but act more like black boxes. Product managers must decide based on specific application scenarios and use cases.
If your product serves a highly regulated industry, legal requirements may demand explainable predictions. For example, healthcare and legal sectors must comply with the EU’s General Data Protection Regulation (GDPR). How should we balance explainability and accuracy?
As a product manager, you must maintain insight and stay aligned with data scientists and engineers from the outset.
Additionally, product managers must consider whether the product has gender or ethical implications: ensuring data diversity and representativeness, or avoiding biases like racial or gender discrimination.
One of my favorite examples is Google Translate. On the left is gender-neutral Turkish, and on the right is the English translation. If you read it, you’ll quickly spot the bias: 'He works hard; she is lazy.' This shows how catastrophic gender bias can infiltrate a product.
You might think of removing gender as a feature, but it’s not that simple. Gender is a critical feature in medical diagnostics—for instance, prostate cancer only occurs in men.
While it may matter less in translation products, it’s highly relevant in HR systems matching job descriptions to candidates. We should provide guidance on handling such biases.
Last December, Google released a version addressing translation bias, offering both masculine and feminine translations for every neutral phrase.
The most challenging issue is that the success factors for machine learning projects differ entirely between research and production environments. The following diagram comes from a famous publication called 'The Hidden Technical Debt in Machine Learning Systems.'
In real-world machine learning systems, the actual machine learning code is just a small component, represented by the small black box in the middle. However, the surrounding infrastructure required is vast and complex. This contradicts common perceptions—you might think machine learning is the largest and most critical component of your AI product, but in reality, it's just a small piece.
In our Salesforce Einstein system, we have a very similar diagram.
Data scientists and data engineers are actually designing a product that needs to answer a variety of questions before it can be deployed in production:
- "Is your data confirmed to be on-premises or in the cloud?"
- "Is your machine learning model confirmed to be on a server, or does it not need to be distributed to mobile devices?"
- "Are you prepared to retrain your machine learning model, and for what reason?"
- "Does your product require real-time predictions? While real-time predictions are highly useful, they are also complex and challenging to design. Alternatively, you might use an offline batch processing system."
Every day, thousands of research papers are published, but few are applied at scale in production. As AI product managers, we must develop sharp insights: determining which products are worth investing in and which can truly be brought to real users.
Although AI is currently trending, it has existed for over 20 years. It was primarily used in search and advertising—like the search suggestions you see on Google, the recommended ads on Yahoo, or the autocorrect when searching on Amazon. If you work in search or advertising, you must provide clear specifications and collaborate closely with data engineers.
Today, the applications of AI are growing exponentially, requiring product managers to provide extensive and clear product specifications. No matter how much your data scientists love their little garden or how creative and cool your marketing demos are, the 'AI product gap' could become a barrier to your AI product's adoption.
As an isolated island, more and more AI products need management. As a product manager, you need to upgrade your skill set to provide valuable specifications for data scientists in your engineering projects.
Finally, answer this question:
When faced with a real, specific customer pain point, can you build a useful product rather than just a cool feature?