Cloud-Agnostic and Lightweight Big Data Processing Platform in Multiple Clouds Using Docker Swarm and Terraform

Big data necessitates the processing of data at scale and efficiency, which is beyond the capabilities of traditional IT infrastructure. Cloud-based big data processing platforms are the most efficient and established infrastructure to accomplish the big data processing requirements. Now organisations are recognising the requirement of a multi-cloud big data processing platform, as it has several benefits such as lower risk, resilience, flexibility, cost-effectiveness, and performance efficiency. However, the orchestration process of a multi-cloud platform is a very complex process due to the interoperability issue between clouds. Therefore, this paper proposes a distinctive application of Docker Swarm in building a cloud-agnostic and lightweight big data processing platform with the assistance of Infrastructure as Code (IaC) tool Terraform. It presents the architectural design and simulated implementation of the proposed big data processing platform in multiple clouds using Docker Swarm. Then, it illustrates the process of making this Docker-based big data platform, a cloud-agnostic and lightweight platform by using Terraform. Subsequently, it demonstrates the reason for using Terraform by comparing it with popular IaC tools. Finally, it performs a comparative evaluation of the proposed containerization based platform with the traditional virtualization based platform to expound the relative efficacy of the Docker Swarm-based big data processing platform.