6MajorIssuesofBigDataSystems·Access patterns are unpredictable- data analytics can be in different ways: Locality may not be a major concern-Every data may be important (e.g. a key word search).Majorconcerns-Scale out: throughput + as the number of nodes +- Fault tolerant (nodes are frequently dead)- Low cost processing for increasingly large volumes: These are not largely considered in existing systems
Major Issues of Big Data Systems • Access patterns are unpredictable – data analytics can be in different ways • Locality may not be a major concern – Every data may be important (e.g. a key word search) • Major concerns – Scale out: throughput + as the number of nodes + – Fault tolerant (nodes are frequently dead) – Low cost processing for increasingly large volumes • These are not largely considered in existing systems 6
MapReduce Data Processing EngineA simple but effective programming model designed toprocess huge volumes of data concurrentlyTwo unique properties- Minimum dependency among tasks (almost sharing nothing)- Simpletaskoperationsineachnode(lowcostmachinesaresufficient). Two strong merits for big data anaytics- Scalability (Amadal's Law): increase throughput byincreasing # of nodes-Fault-tolerance (quick and low cost recovery of thefailuresof tasks)·HadoopisawidelyusedsoftwareofMapReduce- in thousands of society-dependent corporations andorganizations for big data analytics: AOL, Baidu, EBayFacebook,IBM,NY Times,Yahoo!7
MapReduce Data Processing Engine • A simple but effective programming model designed to process huge volumes of data concurrently • Two unique properties – Minimum dependency among tasks (almost sharing nothing) – Simple task operations in each node (low cost machines are sufficient) • Two strong merits for big data anaytics – Scalability (Amadal’s Law): increase throughput by increasing # of nodes – Fault-tolerance (quick and low cost recovery of the failures of tasks) • Hadoop is a widely used software of MapReduce – in thousands of society-dependent corporations and organizations for big data analytics: AOL, Baidu, EBay, Facebook, IBM, NY Times, Yahoo! . 7
MapReduce Operations on HadoopGet average salary of each of 2 organizations in a huge file(name: (org., salary))(org.: avg. salary)KeyValueKeyValueOriginal key/value pairs:Result key/value pairs: twoall the person namesentries showing the org nameassociated with each organd corresponding averagesalaryname and their salariesName(dept.,salary)dept.avg.salaryAlice(Org-1, 3000)Org-1Bob(Org-2, 3500)Org-28
MapReduce Operations on Hadoop • Get average salary of each of 2 organizations in a huge file. {name: (org., salary)} {org.: avg. salary} 8 Key Value Key Value Original key/value pairs: all the person names associated with each org name and their salaries Result key/value pairs: two entries showing the org name and corresponding average salary Name (dept. ,salary) Alice (Org-1, 3000) Bob (Org-2, 3500) . . dept. avg. salary Org-1 . Org-2
MapReduce Operations on Hadoop Calculate the average salary of every organization(name: (org., salary))(org.: avg. salary)HDFSA HDFS blockHadoop Distributed File System (HDFS)
HDFS MapReduce Operations on Hadoop • Calculate the average salary of every organization {name: (org., salary)} {org.: avg. salary} 9 A HDFS block Hadoop Distributed File System (HDFS)
MapReduce Operations on Hadoop: Calculate the average salary of every department(name: (org., salary))(org.: avg. salary]TTDHDFSMapMapMap业业力Each map task takes 4 HbFS blocks as its inputand extract (org. salary? as new key/value pairs,(Alice: (org-1, 3000)o (org-1: 3000)e.g.3 Map tasks concurrently process input dataRecords of "org-1"Records of "org-2"10
HDFS MapReduce Operations on Hadoop • Calculate the average salary of every department Map Map Map {name: (org., salary)} {org.: avg. salary} 10 Each map task takes 4 HDFS blocks as its input and extract {org.: salary} as new key/value pairs, e.g. {Alice: (org-1, 3000)} to {org-1: 3000} 3 Map tasks concurrently process input data Records of “org-1” Records of “org-2