Discriminative Models and Generative Models in Machine Learning

[baoshi.rao]

In machine learning, supervised learning can be divided into two types of models: discriminative models and generative models. Simply put, discriminative models focus on modeling conditional distributions, while generative models target joint distributions.

01 Basic Concepts

Suppose we have training data (X, Y), where X is the set of attributes and Y is the category label. When a new sample x arrives, we want to predict its category y.

Our ultimate goal is to determine the maximum conditional probability P(y|x) as the classification for the new sample.

1. What Discriminative Models Do

Based on the training data, discriminative models derive a classification function and decision boundary. For example, an SVM model produces a decision boundary and directly calculates the conditional probability P(y|x). The class with the highest P(y|x) is selected as the classification for the new sample. Discriminative models model conditional probabilities and learn the optimal boundary between different classes, but they cannot reflect the inherent characteristics of the training data. Their capability is limited—they only tell us the classification category.

2. What Generative Models Do

Generative models typically build a model for each class. For example, if the category labels are {cat, dog, pig}, the model first learns a cat model based on cat features, then a dog model based on dog features, and so on. The joint probability P(y|x) is calculated for the new sample X with each class, and Bayes' theorem is applied:

P(y|x) is computed for each class, and the class with the highest P(y|x) is chosen as the sample's classification.

3. Summary of Both Models

Both generative and discriminative models ultimately rely on the conditional probability P(y|x). However, generative models first compute the joint probability P(x, y) and then derive the conditional probability using Bayes' theorem. As a result, generative models can capture more information about the data distribution and are more versatile.

02 Illustrating Concepts with Examples

1. The Goat vs. Sheep Example

Discriminative Model: To determine whether a sheep is a goat or a sheep, the discriminative model learns from historical data and predicts the probability of the sheep being a goat or a sheep based on its features.

Generative Model: The generative model first learns a goat model based on goat features and a sheep model based on sheep features. The features of the sheep are then evaluated against both models, and the class with the higher probability is selected.

2. Model Examples

Suppose we have a classification problem where X represents features and Y represents class labels. A discriminative model learns a conditional probability distribution P(y|x), while a generative model learns a joint probability distribution P(x, y).

For example, if X has two features (1 or 2) and Y has two classes (0 or 1), with training samples (1, 0), (1, 0), (1, 1), (2, 1), the learned conditional probability distribution (discriminative model) and joint probability distribution (generative model) would differ.

In practical classification problems, discriminative models can directly determine the class of features, while generative models require the application of Bayes' theorem. However, generative models are more general and versatile. Discriminative models are simpler and more straightforward. The two approaches often overlap—generative models can yield discriminative models, but the reverse is not true.

03 Differences Between Discriminative and Generative Models

1. Comparison Diagram

The left side of the diagram shows a discriminative model, while the right side shows a generative model. The key difference is that discriminative models seek a decision boundary to classify samples, whereas generative models learn the boundaries of each class and can generate samples.

2. Characteristics of Both Models

Discriminative Model Characteristics:

1. Models conditional probabilities and learns the optimal boundary between classes.
2. Captures differences between classes but not the data's inherent distribution.
3. Lower computational cost and resource requirements.
4. Effective even with fewer samples.
5. Better performance in prediction.
6. Cannot be converted into a generative model.

Generative Model Characteristics:

1. Models joint probabilities and learns the distribution of all classes.
2. Captures more inherent data characteristics.
3. Higher computational cost and resource requirements.
4. Requires more samples for effective learning.
5. Inferior performance in inference.
6. Can be converted into a discriminative model under certain conditions.

In summary, both models aim to maximize posterior probability. Discriminative models directly model posterior probability, while generative models use Bayes' theorem to transform the problem into computing joint probability.

04 Algorithms for Each Model

(Note: The original content did not provide specific algorithms, so this section is left blank.)