Vancouver, BC, Canada
August 27 & 28 - Co-Located Events, Tutorials, Labs & Lightning Talks
August 29-31 - Conference
Click Here For Information & Registration
Back To Schedule
Thursday, August 30 • 2:00pm - 2:40pm
Building Scalable and Extendable Data Pipelines For Call Of Duty Games: Lessons Learned - Yaroslav Tkachenko, Activision

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct?
We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases.

In this presentation you'll hear about the lessons we learned, including (but not limited to):
- Message schemas
- Apache Kafka organization and tuning
- Topics naming conventions, structure and routing
- Reliable and scalable producers and ingestion layer
- Stream processing

avatar for Yaroslav Tkachenko

Yaroslav Tkachenko

Senior Data Engineer, Activision
Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, functional programming, modern cloud infrastructure and DevOps practices. Currently Yaroslav is a Senior Data Engineer at Activision, working on a large-scale data pipeline. Prior to joining... Read More →

Thursday August 30, 2018 2:00pm - 2:40pm PDT
Room 118