Big Data

QoS-driven Resource Management for Big-data Computing Systems

---

Together with the increasing adoption of cloud computing is the emergence of data-intensive applications. Dubbed as ‘‘big data’’, these applications need to process and analyze massive amounts of data in parallel. As the demand of data-intensive computing continues to grow, it becomes increasingly common to use shared infrastructure to run such applications, while virtualization plays a key role to transparently and flexibly consolidate these applications. However, it is challenging to meet the applications’ Quality of Service (QoS) requirements in such a virtualized data-intensive computing environment. Data-intensive applications require high-performance IOs, relying on distributed storage and network resources which are difficult to allocate in a shared infrastructure. Even if these necessary resource control knobs are available, it is still difficult to determine the necessary allocations because data-intensive applications are highly parallel and intrinsically complex. Finally, the use of virtualized resources adds additional complexity and dynamism to the resource management problem. The objective of this proposed project is to address the resource management challenges for virtualized data-intensive computing and enable diverse applications to meet their desired QoS in such environments.

Participants

• Dr. Ming Zhao (faculty)
• Yiqi Xu (PhD student, VMware Graduate Fellow)
• Saman Biook Aghazadeh (PhD Student)
• Michel Roger (Master student, FIU B.S., past Research Experiences for Undergraduates participant)

Publications

• M. Roger, Y. Xu and M. Zhao, “BigCache for Big-data Systems,” IEEE International Conference on Big Data, October 2014.
• D. Otstott, N. Evans, L. Ionkov, M. Zhao, and M. Lang, “Enabling Composite Applications through an Asynchronous Shared Memory Interface,” IEEE International Conference on Big Data, October 2014.
• Y. Lu, M. Zhao, G. Zhao, L. Wang, N. Rishe, “v-TerraFly: Large Scale Distributed Spatial Data Visualization with Autonomic Resource Management,” Journal Of Big Data, January 2014.
• Y. Lu, M. Zhao, G. Zhao, L. Wang, N. Rishe, “Massive GIS Database System with Autonomic Resource Management,” International Conference on Machine Learning and Applications, December 2013.
• Yiqi Xu, Adrian Suarez, and Ming Zhao, “IBIS: Interposed Big-data I/O Scheduler,” 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC2013), June 2013. (Short Paper)
• J. Quan, Y. Shi, and M. Zhao, “The Implications from Benchmarking Three Big Data Systems,” 1st Workshop on Benchmarks, Performance Optimization, and Emerging Hardware of Big Data Systems and Applications (BPOE, co-held with IEEE BigData 2013), October 2013.

Acknowledgement

 

This material is based upon work supported by a VMware Faculty Fellowship and a VMware Graduate Fellowship and collaboration with Department of Energy Los Alamos National Lab. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.