You train a classification model with 10,000 images of shoes. Everything works wonderfully.
The next Sunday, somebody releases a new pair of sneakers.
Your model has never seen them, but now it needs to classify them somehow.
This suck.
↓
1 of 10
The next Sunday, somebody releases a new pair of sneakers.
Your model has never seen them, but now it needs to classify them somehow.
This suck.
↓
1 of 10
Classification models don't know what they don't know.
If I show you a pair of shoes that you've never seen before, your answer will be, "I don't know what they are."
But a machine learning model doesn't know its limitations.
It will make a (wrong) prediction.
2 of 10
If I show you a pair of shoes that you've never seen before, your answer will be, "I don't know what they are."
But a machine learning model doesn't know its limitations.
It will make a (wrong) prediction.
2 of 10
Of course, this prediction will be wrong.
You didn't train the model with images of this new pair of shoes. You aren't even considering them as a potential valid output!
Still, you'll get back a prediction. A bad one.
3 of 10
You didn't train the model with images of this new pair of shoes. You aren't even considering them as a potential valid output!
Still, you'll get back a prediction. A bad one.
3 of 10
We like to tell ourselves that the model will return a low-confidence prediction when this happens.
Unfortunately, it doesn't work like that.
You could get a high-confidence prediction, especially when your classes are very similar.
4 of 10
Unfortunately, it doesn't work like that.
You could get a high-confidence prediction, especially when your classes are very similar.
4 of 10
I'm writing this because I want to make two separate points:
1. This happens all the time. It sucks, and you need to find ways to understand when you shouldn't act on bad predictions.
2. Your model's performance will start degrading as it makes more mistakes.
5 of 10
1. This happens all the time. It sucks, and you need to find ways to understand when you shouldn't act on bad predictions.
2. Your model's performance will start degrading as it makes more mistakes.
5 of 10
We call this problem "drift."
Many people think machine learning is just like regular software: build it once, and run it until Apple decides to migrate to a different architecture.
It doesn't work like that:
Models require constant babysitting and maintenance.
6 of 10
Many people think machine learning is just like regular software: build it once, and run it until Apple decides to migrate to a different architecture.
It doesn't work like that:
Models require constant babysitting and maintenance.
6 of 10
I'm sure many of you are hearing about "drift" for the first time.
That's a shame.
Machine learning education is hardly concerned with the things that happen after you finish training a model.
They tell you that you are done when training finishes.
7 of 10
That's a shame.
Machine learning education is hardly concerned with the things that happen after you finish training a model.
They tell you that you are done when training finishes.
7 of 10
To be fair, some courses talk about concept and data drift, but most don't.
And the vast majority of those who mention it do it casually and spend zero time on it.
Raise your hand if you received formal training on detecting and combat drift.
That's what I thought.
8 of 10
And the vast majority of those who mention it do it casually and spend zero time on it.
Raise your hand if you received formal training on detecting and combat drift.
That's what I thought.
8 of 10
Drift causes your model's performance to suck a little bit more every day.
The antidote is monitoring and maintenance.
This is not sexy and doesn't make headlines, but it's what the work is about.
9 of 10
The antidote is monitoring and maintenance.
This is not sexy and doesn't make headlines, but it's what the work is about.
9 of 10
If you read this far, and you want something actionable from this thread, go and read about:
• Concept drift
• Data drift
• Machine learning monitoring
Building your model is just the first milestone, but hardly the last one.
10 of 10
• Concept drift
• Data drift
• Machine learning monitoring
Building your model is just the first milestone, but hardly the last one.
10 of 10
Follow me @svpino and join the other 149,000+ people who show up for my writing on practical machine learning.
Every day I break down complex concepts and help you navigate the industry.
Appreciate your support!
Every day I break down complex concepts and help you navigate the industry.
Appreciate your support!
Loading suggestions...