Migrating a Large Scale Search Dataset in Production in a Highly Available Manner

Shopify is an ecommerce platform supporting over 3 million global merchants which uses Google Kubernetes Engine to run Elasticsearch on Google Cloud Platform. The COVID-19 pandemic led to an increase in global clients, causing latency issues and GDPR compliance challenges. To address this, the search infrastructure team at Shopify migrated European merchants’ search data to European regions. However, this migration was complex due to the mixed storage of European and non-European merchants’ data and the constraints of the indexing pipeline. Moreover, the scale of data that needed to be migrated was large and would lead to outages for merchants’ search services which would negatively impact their revenue. This talk tells the story of how this team designed an architecture to migrate a large dataset to European regions without impacting merchants’ sales. We will review the technical decisions and the tradeoffs that were made to address the challenges faced.