Pinterest is a personalized data driven product that serves hundreds millions users. To this end, engineers and analytics need to collect data, extract features, build signals, train machine learning models, do ETLs, generate a/b dashboard and compute business metrics. Job orchestration and scheduling system is a fundamental need: there needs a way for connecting each small dots into necklaces.
In this talk, Shuo and Jack will present how Pinterest solved this problem using their open source workflow management system called Pinball (
https://github.com/pinterest/pinball). More specifically, they will share the pain points, why and how they built Pinball and make it a scalable/flexible ecosystem to cover use cases from dashboard generating and index building to complicated machine learning model training.
