IMO there's no substitute MLOps experience for building a pipeline that serves predictions at some endpoint (e.g., REST) and trying to sustain some performance over time. Some pointers & tutorials below:
1. Convince yourself that operationalizing ML, even as a 1-person team, is a hard problem. What are some differences between a kaggle project and a production ML service? Do some tutorials -- Here's a more-than-hello-world toy ML pipeline I've built: github.com
Here's a more in-depth tutorial that simulates some performance degradation/drift: github.com
2. Understand that Ops is also, largely, a people problem. This is probably hard to do alone or w/o company experience. I like laszlo.substack.com
3. Get familiar with standard tools that are used across various companies to solve MLOps problems:
Basics: Postgres, some DAG scheduler (e.g., Airflow), dashboards for metrics (e.g., Grafana, Metabase), matplotlib, experiment tracking (e.g., mlflow), assert statements
Basics: Postgres, some DAG scheduler (e.g., Airflow), dashboards for metrics (e.g., Grafana, Metabase), matplotlib, experiment tracking (e.g., mlflow), assert statements
4. Learn what to monitor and how to monitor:
What: percentiles & means of features & outputs, univariate testing for features & outputs (e.g., KS test)
How: Prometheus, Cloudwatch, Datadog, idk some way to emit some value to a table you can quickly aggregate over
What: percentiles & means of features & outputs, univariate testing for features & outputs (e.g., KS test)
How: Prometheus, Cloudwatch, Datadog, idk some way to emit some value to a table you can quickly aggregate over
5. There's so much interesting content out there on operationalizing DS and ML. Some of my favorite writers in the space are @vboykis @chipro @eugeneyan @xLaszlo -- go read what they write!!!
Loading suggestions...