(opinionated) thoughtdump of industry vs academic "mlops" tool-building philosophies (writing this more for myself as a means of clarifying my thoughts, and all of this is merely speculation based on using tools and reading papers): 1/8
industry-originated tools skew towards heavy lock-in / buy-in from users (i.e. code rewrites); academic-originated tools skew towards little-to-no code modifications (i.e. instrument the python AST to track provenance) 2/8
industry-originated tools seem to prioritize logging as a means to reproducibility (i.e. force users to write computations in DAGs upfront); academic-originated tools seem to prioritize *what* one can do with logging (i.e. debugging guidance) 3/8
industry-originated tools seem to prioritize collaboration / multiple people working on the same workflow (i.e. logging to remote server by default, and some tools are even k8s only); academic-originated tools seem to camp out in the machine that runs the computation 4/8
few understand "end-to-end." in my opinion, end-to-end means literally, one end to another end. clean data is not an end. features are not an end. a trained model is not an end. the most upstream source of data is one end, and the output shown to a client is the other end 5/8
many tools don't really do all the things they claim to do, or they might do a crappy job of it. this is not really a *bad* thing. i don't think this is specific to the field -- it's probably a consequence of the free market 6/8
the most successful tools seem to have big sales / marketing efforts and require lock-in (some codebase changes). i guess this is how B2B works (ugh). but this means there is room to induce paradigm shifts in developing, deploying, and maintaining workflows 7/8
the challenge is: how radical can the change be. i.e. personally i believe no DS should need to write yaml but what is really a better alternative without reinventing schedulers or adding a layer that only transpiles & complicates the stack further. fun thoughts to 🍜 on 8/8
i feel the need to clarify that this was intended to be nonjudgemental. there are tradeoffs everywhere, and there's no one "correct" philosophy imo (9/8)
Loading suggestions...