I am a Senior Software Engineer in the Audience Platform Data team responsible for processing and analysing the billions of messages audiences create when interacting with BBC online services. Previously I worked with XML based technologies on the BBC Red Button Digital Text service.
The BBC Audience Platform Data team collects, transforms and delivers billions of events each day from audience interactions with mobile apps and web sites such as BBC News, BBC Sport and iPlayer.
We will share our experience of the 7 critical steps we took to migrate from Spark clusters with Map-Reduce to a serverless architecture. We will cover the good, the bad and the ugly, including lessons learnt along the way. These steps are also applicable to new architectures.
We have been running in production for nearly a year and have achieved our goal of real-time analysis at up to 25000 transactions per second. This allows us to meet the demands of even the biggest news days, as well as seeing significant savings on our cloud infrastructure.