Machine Learning vs Statistics

Insightful comment on HN from 2017 (slightly paraphrased): https://news.ycombinator.com/item?id=14944613

Machine Learning: Everything that focuses on dealing with problems with a complex structure and low noise

Statistics: Everything that focuses on dealing with problems with a large amount of noise

The whole comment is interesting, especially this part:

Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The “noise” is in discovering the unknown relationship between pixels and class, not in the actual output.

Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it’s generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit.

I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise.

Incidentally, this was perhaps the inspiration for AI Snake Oil - Aravind Narayanan per https://news.ycombinator.com/item?id=45306256

most people don’t appreciate the distinction between domains with low irreducible error that benefit from fancy models with complex decision boundaries (like computer vision) and domains with high irreducible error where such models don’t add much value over something simple like logistic regression