Open Access iconOpen Access

ARTICLE

crossmark

Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN

J. S. Manjaly, T. Subbulakshmi*

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, 600127, India

* Corresponding Author: T. Subbulakshmi. Email: email

Computer Systems Science and Engineering 2023, 47(3), 3083-3108. https://doi.org/10.32604/csse.2023.036320

Abstract

The default scheduler of Apache Hadoop demonstrates operational inefficiencies when connecting external sources and processing transformation jobs. This paper has proposed a novel scheduler for enhancement of the performance of the Hadoop Yet Another Resource Negotiator (YARN) scheduler, called the Adaptive Node and Container Aware Scheduler (ANACRAC), that aligns cluster resources to the demands of the applications in the real world. The approach performs to leverage the user-provided configurations as a unique design to apportion nodes, or containers within the nodes, to application thresholds. Additionally, it provides the flexibility to the applications for selecting and choosing which node’s resources they want to manage and adds limits to prevent threshold breaches by adding additional jobs as needed. Node or container awareness can be utilized individually or in combination to increase efficiency. On top of this, the resource availability within the node and containers can also be investigated. This paper also focuses on the elasticity of the containers and self-adaptiveness depending on the job type. The results proved that 15%–20% performance improvement was achieved compared with the node and container awareness feature of the ANACRAC. It has been validated that this ANACRAC scheduler demonstrates a 70%–90% performance improvement compared with the default Fair scheduler. Experimental results also demonstrated the success of the enhancement and a performance improvement in the range of 60% to 200% when applications were connected with external interfaces and high workloads.

Keywords


Cite This Article

APA Style
Manjaly, J.S., Subbulakshmi, T. (2023). Performance improvement through novel adaptive node and container aware scheduler with resource availability control in hadoop YARN. Computer Systems Science and Engineering, 47(3), 3083-3108. https://doi.org/10.32604/csse.2023.036320
Vancouver Style
Manjaly JS, Subbulakshmi T. Performance improvement through novel adaptive node and container aware scheduler with resource availability control in hadoop YARN. Comput Syst Sci Eng. 2023;47(3):3083-3108 https://doi.org/10.32604/csse.2023.036320
IEEE Style
J.S. Manjaly and T. Subbulakshmi, "Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN," Comput. Syst. Sci. Eng., vol. 47, no. 3, pp. 3083-3108. 2023. https://doi.org/10.32604/csse.2023.036320



cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 338

    View

  • 227

    Download

  • 0

    Like

Share Link