Vancouver, BC, Canada
August 27 & 28 - Co-Located Events, Tutorials, Labs & Lightning Talks
August 29-31 - Conference
Click Here For Information & Registration
Thursday, August 30 • 2:00pm - 2:40pm
Building Scalable and Extendable Data Pipelines For Call Of Duty Games: Lessons Learned - Yaroslav Tkachenko, Activision

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct?
We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases.

In this presentation you'll hear about the lessons we learned, including (but not limited to):
- Message schemas
- Apache Kafka organization and tuning
- Topics naming conventions, structure and routing
- Reliable and scalable producers and ingestion layer
- Stream processing

avatar for Yaroslav Tkachenko

Yaroslav Tkachenko

Senior Data Engineer, Activision
Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, functional programming, modern cloud infrastructure and DevOps practices. Currently Yaroslav is a Senior Data Engineer at Activision, working on a large-scale data pipeline. Prior to joining... Read More →

Thursday August 30, 2018 2:00pm - 2:40pm
Room 118