國立虎尾科技大學 |

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks./
Author:	Abusalah, Bara.
Description:	1 online resource (104 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-01, Section: B.
Contained By:	Dissertations Abstracts International85-01B.
Subject:	End users. -
Online resource:	click for full text (PQDT)
ISBN:	9798379829179

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
Abusalah, Bara.

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks. - 1 online resource (104 pages)

Source: Dissertations Abstracts International, Volume: 85-01, Section: B.

Thesis (Ph.D.)--Purdue University, 2021.

Includes bibliographical references

The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of new Big Data frameworks every single year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka ... etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. Despite all the advances in making datacenters dependable, failures actually still happen. This is particularly onerous for long-running "big data" applications, where partial failures can lead to significant losses and lengthy recomputations. This is also crucial for streaming systems where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users.Another observation is that some reliability implementations are redundant between different frameworks. Big data processing frameworks like Hadoop MapReduce include fault tolerance mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. Encapsulating these implementations into one layer and making it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework.These observations motivated us to solve the problem by presenting two systems: Guardian and Warden. Guardian is tailored towards batch processing big data systems while Warden is targeted towards stream processing systems. Both systems are robust, RMS based, generic, multi-framework, flexible, customizable, low overhead systems that allow their users to run their applications with individually configurable fault tolerance granularity and degree, with only minor changes to their implementation.Most reliability approaches carry out one rigid fault tolerance technique targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks in any part of the application code. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two fault tolerance guarantees (Crash Failures / Byzantine Failures) and, in case of streaming systems; it is combined with two delivery semantics (Exactly Once / At Most Once).In other words, this thesis proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing fault tolerance support at the level of such an RMS yields generic fault tolerance mechanisms, which can be provided with low overhead by leveraging constraints on resources.To the best of our knowledge, such approach was never tried on multiple big data batch processing and streaming frameworks before.We demonstrate the benefits of Guardian by evaluating some batch processing frameworks such as Hadoop, Tez, Spark and Pig on a prototype of Guardian running on AmazonEC2, improving completion time by around 68% in the presence of failures, while maintaining around 6% overhead. We've also built a prototype of Warden on the Flink and Samza (with Kafka) streaming frameworks. Our evaluations on Warden highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798379829179Subjects--Topical Terms:

1472953
End users.
Index Terms--Genre/Form:

554714
Electronic books.

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
LDR:05275ntm a22003257 4500 001 1147268
005 20240909100741.5
006 m o d
007 cr bn ---uuuuu
008 250605s2021 xx obm 000 0 eng d
020 $a 9798379829179
035 $a (MiAaPQ)AAI30504653
035 $a (MiAaPQ)Purdue14477733
035 $a AAI30504653
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Abusalah, Bara. $3 1472952
245 1 0 $a Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
264 0 $c 2021
300 $a 1 online resource (104 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-01, Section: B.
500 $a Advisor: Ghafoor, Arif.
502 $a Thesis (Ph.D.)--Purdue University, 2021.
504 $a Includes bibliographical references
520 $a The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of new Big Data frameworks every single year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka ... etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. Despite all the advances in making datacenters dependable, failures actually still happen. This is particularly onerous for long-running "big data" applications, where partial failures can lead to significant losses and lengthy recomputations. This is also crucial for streaming systems where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users.Another observation is that some reliability implementations are redundant between different frameworks. Big data processing frameworks like Hadoop MapReduce include fault tolerance mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. Encapsulating these implementations into one layer and making it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework.These observations motivated us to solve the problem by presenting two systems: Guardian and Warden. Guardian is tailored towards batch processing big data systems while Warden is targeted towards stream processing systems. Both systems are robust, RMS based, generic, multi-framework, flexible, customizable, low overhead systems that allow their users to run their applications with individually configurable fault tolerance granularity and degree, with only minor changes to their implementation.Most reliability approaches carry out one rigid fault tolerance technique targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks in any part of the application code. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two fault tolerance guarantees (Crash Failures / Byzantine Failures) and, in case of streaming systems; it is combined with two delivery semantics (Exactly Once / At Most Once).In other words, this thesis proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing fault tolerance support at the level of such an RMS yields generic fault tolerance mechanisms, which can be provided with low overhead by leveraging constraints on resources.To the best of our knowledge, such approach was never tried on multiple big data batch processing and streaming frameworks before.We demonstrate the benefits of Guardian by evaluating some batch processing frameworks such as Hadoop, Tez, Spark and Pig on a prototype of Guardian running on AmazonEC2, improving completion time by around 68% in the presence of failures, while maintaining around 6% overhead. We've also built a prototype of Warden on the Flink and Samza (with Kafka) streaming frameworks. Our evaluations on Warden highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a End users. $3 1472953
650 4 $a Data processing. $2 sears $3 840972
650 4 $a Failure. $3 1365603
650 4 $a Computer science. $3 573171
650 4 $a Batch processing. $3 1472954
650 4 $a Fault tolerance. $3 1472955
650 4 $a Cloud computing. $3 716205
650 4 $a Real time. $3 1465795
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Purdue University. $3 1184550
773 0 $t Dissertations Abstracts International $g 85-01B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30504653 $z click for full text (PQDT)