📄 How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation?

Abstract

This workshop paper compares the accuracy of energy estimation approaches that use resource utilization and linear power models with Intel RAPL energy measurements for Apache Spark applications running on Kubernetes within an AWS bare-metal cloud and an on-premises cluster. The study demonstrates that adding external monitoring signals improves estimation accuracy relative to Spark task traces.

A preprint of the paper is available on arXiv and an artifact (raw results and prototype code) is available on Zenodo.