u/Jwelch25in r/dataengineering • Recommends
Data-Ops-ish Design and Workflow Walkthrough
In another thread, u/brendersplide asked about building a regression test suite. In response to my comment, there were a few individuals who wanted to know more about the system I am currently running. ( u/isleepbad and u/htrp) It's more than what should be in a reply so I thought I would do a post for anyone else interested.
WARNING: This is a long post.
DISCLAIMER: This is, by no means, the best way to do things. We are constantly evolving and tweaking.
As a little context, I run data and analytics at a start-up-ish financial company and have a very small team. In order to combat that constraint, we have done a lot to automate our workflow.…