This is quite interesting. I am a huge champion of "iterate on the data, not the model." However after further thought, I think there is actually greater opportunity to create academic benchmarks that actually hit the nail on the head for industry ML problems: (1/7)
In the thread @AndrewYNg (accurately) argues that iterating on the data is more fruitful in developing ML applications. This begs the question, what really is the biggest difference between academic ML and industry ML tasks? (2/7)
I think the biggest difference is that academic ML tasks (generally) are evaluated on a constant, fixed test set whereas industry ML tasks need to be applied on many groups of data points over longer periods of time. So in industry ML, the lowest hanging fruit is the data. (3/7)
A new benchmark where the training algorithm is held constant definitely motivates people to focus more on the data, but if the evaluation set is held constant, I would suspect that the top solutions are again "Netflix Prize"-esque solutions (hard to productionize). (4/7)
I'd guess that the best solution would include a large amount of one-off record dropping, too many human passes over the training data, different interpolation techniques for each column, etc -- to the point where this process not easily replicable for another task. (5/7)
The goal is to make benchmarks & competitions that most accurately mimic the problems we try to solve. An ideal benchmark (IMO) is a task where the developer or researcher is forced to deploy training/retraining policies over long periods of time where the data is changing. (6/7)
We don't even need to require that the training algo is held constant; in fact, we shouldn't b/c we should prioritize solutions that work best for the problems we are trying to solve. I imagine initially the solutions will be data-centric; but that may change over time. (7/7)
Loading suggestions...