A Configurable Rule-Based Classful Token Bucket Filter
Network Request Scheduler for the Lustre File System
SessionCutting Edge File Systems
Event Type
Paper
File Systems
QoS
Storage
TimeTuesday, November 14th11:30am -
12pm
Location405-406-407
DescriptionHPC file systems today work in a best-effort manner
where individual applications can flood the file system
with requests, effectively leading to a denial of
service for all other tasks. This paper presents a Token
Bucket Filter (TBF) policy for the Lustre file system.
The TBF enforces RPC rate limitations based on
(potentially complex) Quality of Service (QoS) rules.
The QoS rules are enforced in Lustre's Object Storage
Servers, where each request is assigned to an
automatically created QoS class.
The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.
Download PDF:
here
