AI Development Guide: What is a Machine Learning Product?
-
Why is managing machine learning (ML) products more challenging than regular software? The key lies in 'experimentation'!
Here are the most important aspects I consider when managing AI products:
In the article AI Redefining Robotics, I mentioned that the biggest difference ML brings is:
Shifting machines from relying on manually written programs to genuine autonomous learning.
Machines no longer require human instructions but instead make predictions and improvements based on patterns identified in data. This is why ML is particularly suitable for problems that were previously difficult to define explicitly. It also means ML can make your products more personalized, automated, and precise.
Advanced algorithms, big data, and declining hardware costs are the main drivers of ML growth.
AI is gradually being adopted across various fields. A recent McKinsey report shows that nearly half of companies have integrated AI into their R&D processes, with another 30% experimenting with AI projects.
It's not hard to see why many expect ML to bring even greater industrial transformation than mobile technology. However, at the same time, the difficulty of adopting ML for companies may be several times higher than when they adopted mobile technology. Why? Before discussing the specific reasons, let's first talk about what ML is.
There is no universally agreed-upon definition of AI, and its definition is constantly evolving. Once a task can be successfully performed by a machine, it is no longer considered part of AI.
ML is a subset of AI. Carnegie Mellon professor Tom M. Mitchell defines machine learning as an algorithm that 'allows a program to automatically improve through experience.'
There are three main types of machine learning:
Depending on the type of product and the source of its core value, you will need different skills and focus on different aspects of the product.
Consumer ML products like Alexa or Google Assistant have a stronger social interaction component. Therefore, user experience (UX) plays a critical role in designing consumer ML products, while ML technology is often one of the ways to achieve better UX.
For example, NLP (natural language processing) is used to enable more natural communication between Alexa and users. On the other hand, B2B ML products target enterprises or even industrial users (e.g., algorithms for predicting factory equipment maintenance schedules), where the core value often lies in prediction accuracy rather than UX.
This isn't to say UX is unimportant for enterprise-facing ML products. Rather, when resources are limited and you need to focus on optimizing certain aspects of the product, this consideration becomes necessary.
If your product's core value comes from an ML model, you are likely designing an ML product. Conversely, if ML is only used to enhance UX or partial performance, you are likely applying ML to a product.
In the latter case, as a product manager, you shouldn't spend too much time worrying about technical details like 'whether the ML model is based on CNN (convolutional neural networks) or R-CNN.' Instead, focus on understanding the model's inputs and outputs.
For example, does the ML model you're using take user demographic data (input) to predict their monthly spending on the platform (output)? On the other hand, designing ML products typically requires PMs to have higher technical capabilities to help the team make key decisions and trade-offs.
Product type also affects organizational structure. Companies developing ML products or large companies like Facebook and Google, which invest heavily in ML, typically hire ML researchers or data scientists and form teams with ML engineers.
Conversely, for companies looking to apply ML to their products or smaller companies with limited resources, the best strategy is to hire cross-disciplinary ML engineers or train software engineers in ML rather than hiring ML researchers.
Developing ML products rarely involves only ML. It is often interdisciplinary, encompassing not just ML model design and training but also software engineering, backend architecture, data analysis, UX/UI design, and even hardware-software integration.
Product managers need to manage cross-functional teams and handle interdependencies and potential conflicts. ML is fundamentally different from other disciplines, as further explained in the next section. If you're designing ML products that interact with the real world (e.g., robotics or self-driving cars), the situation becomes even more complex.
PMs need to know what ML can and cannot do, and when to use or avoid it.
Overfitting is a common issue that occurs when an ML model is too closely tailored to a specific dataset. A reliable ML model performs well not only on the training dataset but also on the validation dataset. However, in cases of overfitting, performance on training data improves, but performance on unseen validation data worsens.
Deep learning (DL) is primarily used for image classification. DL employs deep neural networks with labeled images as input. Each layer of the network transforms the input into slightly more abstract and composite representations. Eventually, the model learns to recognize the content of the images.
Natural language processing (NLP) is a field of computer science aimed at enabling machines to understand human language, though it doesn't necessarily involve ML. NLP is often used in chatbots, voice assistants, or data preprocessing.
ML also involves code and data, but assuming it's the same as software engineering is a big mistake.
Unlike software engineering, developing ML products requires more experimentation, involving greater uncertainty and variability. Software engineering is a deterministic process of writing rules for machines, while ML is more probabilistic because it learns on its own without explicit rule-writing.
For example, if you want to teach a machine to recognize cats, software engineering might involve defining explicit rules like 'a cat has four legs and two pointy ears.'
But with deep learning, you don't provide explicit rules.
Instead, you give the machine a bunch of labeled cat photos and let it learn and derive the rules on its own.
Your team's job is to define the problem, prepare the data, build the ML model, test and iterate until you have a model that delivers the desired results.
This is why developing ML products often involves higher risks. For PMs, it's important to set the right expectations for the team to avoid potential conflicts.
For instance, software engineers might feel the ML team hasn't provided clear enough requirements. But this isn't necessarily the ML team's fault—during experimentation, even they may struggle to predict the model's final performance. It's crucial to help other teams understand the experimental nature of ML products.
It's also vital for engineers, researchers, and data scientists to collaborate closely to balance priorities and keep the product focused. More importantly, it's best to develop a testable product early and test it regularly to ensure the ML team's algorithms align with product goals.
As mentioned earlier, ML is well-suited for solving complex problems that humans can't define explicitly. Models need training, testing, and tuning. Often, data scientists must test several methods before settling on a satisfactory one. This makes defining milestones and timelines for ML products particularly challenging.
Thus, for PMs, clearly defining requirements, setting success criteria, and ensuring the team frequently tests ML models against those criteria are all critical.
Because ML is fundamentally different from software engineering, it requires organizational changes: embracing an experimental culture, data-driven thinking, and tolerance for uncertainty.
If machine learning is treated as a purely technical issue while neglecting related organizational changes, companies are likely to face what is known as "The Innovator’s Dilemma." This is particularly challenging for hardware companies like robotics manufacturers, which have traditionally pursued high precision. However, while ML improves with more data, it often cannot achieve 100% accuracy from the outset. Additionally, machine learning products require vast amounts of data, so businesses must establish their own data pipelines and infrastructure to support ML product scaling. For most companies, this is also an entirely new challenge.
The term "software engineering" first appeared in 1965, about 15 years after programming languages emerged. It took roughly another 20 years for software engineering institutes to be established to manage software development processes. Today, we have identified best practices for software engineering.
In contrast, machine learning only began to flourish as a distinct field in the 1990s. Deep learning, a subset of ML, has set new records in areas like image recognition and NLP, but it wasn’t widely discussed until after the emergence of AlexNet in 2012.
Compared to software engineering, ML is still in its infancy, lacking industry standards, metrics, infrastructure, and development tools. As a result, businesses are still exploring best practices and popular applications.
Machine learning algorithms function like black boxes—they take inputs (e.g., images) and output predictions (e.g., identifying objects in the image). This makes it difficult for product managers to explain how ML models work and to gain full support from users and stakeholders.
Especially in critical fields like healthcare, accountability and transparency are paramount. Ensuring alignment between ML models and product goals without a clear understanding of how the algorithms actually work is a significant challenge.
How should we manage machine learning products in the face of these challenges? In Part 2, I will share the best practices I’ve learned.