Building Big Data Processing Systemsbased on Scale-Out Computing Models
1 Building Big Data Processing Systems based on Scale-Out Computing Models
Small Data: Locality of ReferencesPrinciple of Locality- A small set of data that are frequently accessed temporally and spatially- Keeping it close to the processing unit is critical for performanceOneoflimitedprinciples/lawsincomputerscienceWhere can weget locality?- Everywhereincomputing:architecture,softwaresystems,applicationsFoundations of exploiting locality-Locality-awarearchitecture-Locality-awaresystemsLocalitypredictionfromaccesspatterns2
Small Data: Locality of References • Principle of Locality – A small set of data that are frequently accessed temporally and spatially – Keeping it close to the processing unit is critical for performance – One of limited principles/laws in computer science • Where can we get locality? – Everywhere in computing: architecture, software systems, applications • Foundations of exploiting locality – Locality-aware architecture – Locality-aware systems – Locality prediction from access patterns 2
Conventional Databases: Move data to compute. Centralized control to achieve ACID- Atomicity: if one part of the transaction fails, the entire onefails- Consistency: from one valid state to another valid state- lsolation: Resource sharing is not allowed- Durability: once a transaction is committed, the resultsneed to be permanently stored.: A centralized approach (or a scale up)- Scale-out: throughput increases as the # nodes increases. A vender controlled technical/business model- Expensive (designed for banks and high profit orgs)- ACID may not be required for massive data processing3
Conventional Databases: Move data to compute • Centralized control to achieve ACID – Atomicity: if one part of the transaction fails, the entire one fails – Consistency: from one valid state to another valid state – Isolation: Resource sharing is not allowed – Durability: once a transaction is committed, the results need to be permanently stored. • A centralized approach (or a scale up) – Scale-out: throughput increases as the # nodes increases • A vender controlled technical/business model – Expensive (designed for banks and high profit orgs) – ACID may not be required for massive data processing 3
How to handle increasingly large volume data? Anew paradigm (from Ivy League to Land Grant model)- 150+ years ago, Europe ended the industrial revolution- But US was a backwardagriculture country- Higher education is the foundation to become a strongindustrialcountry. Extending thelvyLeaguesto massively accept students? Impossible!.Anew higher education model? Must be! Land grant university model: at a low cost and be scalable- Lincoln singed the“Land Grant UniversityBill"in 1862Togivefederal landtomanyStatestobuildpublicuniversities- The missionis to build low costuniversities and open to massesThe success of land grant universities-Althoughthemodelislowcostandlessselectiveinadmissions,theexcellenceofeducationremains- Manyworld class universities wereborn and established bythismodel:Cornel,MiT,OhioState,Purdue,UCBerkeley,UluC,Wisconsin..4
How to handle increasingly large volume data? • A new paradigm (from Ivy League to Land Grant model) – 150+ years ago, Europe ended the industrial revolution – But US was a backward agriculture country – Higher education is the foundation to become a strong industrial country • Extending the Ivy Leagues to massively accept students? Impossible! • A new higher education model? Must be! • Land grant university model: at a low cost and be scalable – Lincoln singed the “Land Grant University Bill” in 1862 – To give federal land to many States to build public universities – The mission is to build low cost universities and open to masses • The success of land grant universities – Although the model is low cost and less selective in admissions, the excellence of education remains – Many world class universities were born and established by this model: Cornel, MIT, Ohio State, Purdue, UC Berkeley, UIUC, Wisconsin . 4
5MajorDifferencesinNewInfrastructure:Shared with conventional databases- SQLcontinues- Enterprise data warehouse (EDW)frameworkcontinues- Other commonly used API and standards (e.g. JDBC, ODBC).Majordifferences-A scale-out computing model (e.g. MapReduce)-Commodity computing and storage systems based-Scale up software efforts and advanced hardwareacceleration are additional efforts-Affordability is a requirement-Community driven open source software
Major Differences in New Infrastructure • Shared with conventional databases – SQL continues – Enterprise data warehouse (EDW) framework continues – Other commonly used API and standards (e.g. JDBC, ODBC) • Major differences – A scale-out computing model (e.g. MapReduce) – Commodity computing and storage systems based – Scale up software efforts and advanced hardware acceleration are additional efforts – Affordability is a requirement – Community driven open source software 5