Multiple Stream Job Performance Optimization with Source
Operator Graph Transformations
Author/Presenter
Event Type
Workshop
Applications
Clouds and Distributed Computing
SIGHPC Workshop
TimeSunday, November 12th11:10am -
11:40am
Location507
DescriptionMultiple distributed stream queries which are executed
on stream processing systems need to be fine tuned to
the compute cluster in order to harness the full
potential of the hardware they run on. In this paper we
describe an automatic technique for conducting such
stream query optimization in the presence of multiple
stream jobs. During this auto-tuning process we identify
the structure of each program and conduct automatic
program transformation to generate optimized unified
streaming jobs. The operators on the unified secondary
sample application are grouped into PEs considering
their performance characteristics and the stream graph
topology structure to produce high performance stream
query network. We implemented this multiple stream query
optimization technique on a mechanism called Tahitica.
We demonstrate our approach's ability for producing
optimized stream query performance by comparing it to
naive deployments using two real world stream processing
applications in the domains of healthcare and search
advertising. Our stream query optimization approach
reported 7.1% throughput performance improvement
compared to a naive deployment.
Author/Presenter
