Why high bias and high variance?

In Machine Learning Modeling, overfitting models have high variance and underfitting models have high bias.

In the context of machine learning, the terms "variance" and "bias" are used to describe the two primary sources of error in a model. These concepts come from the "Bias-Variance Tradeoff," which is a fundamental aspect of model complexity and generalization.

1. Variance: Variance refers to a model's sensitivity to small fluctuations in the training data. High variance models are complex, capturing the noise in the data along with the underlying patterns. As a result, they tend to perform well on the training data but poorly on unseen data (test or validation data), which is a sign of overfitting. Overfitting models are said to have high variance because their performance varies significantly depending on the specific training data used.

2. Bias: Bias refers to a model's inability to accurately capture the underlying patterns in the data due to its simplicity. High bias models make strong assumptions about the data, which may not be accurate, leading to systematic errors. Underfitting models are said to have high bias because they do not capture the true relationship between input features and target variable, resulting in a consistent error pattern. These models perform poorly on both the training and test data.

In practice, there is often a tradeoff between bias and variance, as reducing one typically increases the other. A good model will strike a balance between the two, minimizing the total error and providing the best generalization to new, unseen data.