The paper is a must-read for anyone trying to do ML in production. Want us to give a talk to your group/org? Email shreyashankar@berkeley.edu. You can read the paper for the war stories & insights, so I’ll do a “behind the scenes” & “fave quotes” in this thread instead.
Behind-the-scenes: another school invited my advisor to contribute to a repo of MLOps resources. We contributed what we could, but felt oddly disappointed by the little evidence we could point to for support.
We thought it might be valuable to do what the industry can’t do: an academic analysis of MLOps. Prior academic interview studies broadly study data science & analysis, not MLOps. We wanted to similarly bring theory and rigor to the fragmented MLOps landscape.
You may wish to skip to Section 4 for insights. Here, we tried to write about common practices that weren’t obvious but still generalizable, approachable for beginners yet insightful for veterans, & subtly challenge popular incorrect advice.
Fun quote: “You can’t hit pedestrians, right. You can’t hit cyclists...what you need to be able to do in a mature MLOps pipeline is go very quickly from user recorded bug...to be able to drive improvements to the stack by changing your data based on those bugs.”
Fun quote: “We have hard-coded rules for mission critical customers...we're not just shipping off fake wins, like we're really in the value business.”
Personally, I’ve also seen so many MLOps tools out there that just…don’t solve the right problems, or solve at the wrong layer of abstraction. It is hard to reason why. We wrote Section 5 to shed light on possible pitfalls and tooling opportunities.
Fun quote: "But as for the product, metrics keep on rotating based on the company's priorities, you know."
Fun quote: “If I focus on one idea, a week at a time, then it boosts my productivity a lot more.”
I’m lucky to have been able to work with fellow DB PhD student @rogarcia_sanz (who also has a minor in qualitative methods!) and seasoned HCI & DB profs/writers/founders @adityagp & @joe_hellerstein. I don't think this paper, as is, could've been done anywhere else 😊 Enjoy!
Loading suggestions...