Its old design pattern which is more relevant now. You can
have different parts of calculation run on different machines which can talk to
each other and hence run calculation faster.
Google have their own implementation which is similar to Cloudera implementation. Then Amazon has Hadoop Lite , houch DB or mongo DB have some implementation running inside. Each one is solving specific probs.
MapReduce is basically made up of MAP and REDUCE. Where MAP
is Stateless and independent and REDUCE for it we need to think of how it fits
into overall flow... so there is seperation. Let's think of sorting prob each
node needs to sort its data and fit it into Reducer.
Let's suppose a cluster of N nodes having data and running a
piece of code F(x) (like sum , avg etc) . Now there is no separation of data ex it's not the case one
node has Sept data other Oct node. not data is separated by Product type etc ,
data is evenly distributed. AND DATA is independent.
Each of the nodes will output pairs. can be date
and dollar amount(sales).
Seems Can have more than one reducer. All the key value
pairs come to this Reducer and gives you any key value pair ie sales on any
given day.
Can do similar stuff
in SQL , so whats the advtg here ? First this is not running on enterprise architecture think in terms of clustered architecture. And secondly Advtg here is scalability ... independent
portions don't have to wait for anyone to finish.
Example :- suppose 6 nodes and we have 2 reducers one for
1st quarter Jan - Mar and another for second march so each reducers gets data
from all of the nodes.
Map is like a Data Warehouse code which extracts , Transform
and loads.
It's not necessary to always write/need reduce , 6 out of 10
times you can pass by the reduce code.
Can also chain map reduce.
Icremental Map Reduce :- Couch DB does this once you ask it
a ques ( avg sale this quarter ) it keeps computing this for you. this makes it
different from google. Couch DB is interested in 1 to 100 DB server and Google
is interested in PeraByte plus info like how to scale the web. COuch DB is a doc based database where docs
are represented as JSON when intreacting with it , else stored as binary.
Suppose i want to find biggest sales. so sorting on each
node. sort is done according to key so you need to put the right thing in the
key. so basically ask yourself how do i want to query this thing later on ...
that defines the key.
you can do lot of diff things with Emit(amount,null) if date
is of june then emit if date is in last 30 days then emit else not.
Things like count , sum , min , max , avg , std dev etc are
written to the DB.
Ex :- we want to know the total no of sales of
certain amount. so basically in mapper we return k,v where k should be amount
.. in the mapper code we can have logic if greater than x then return k,v =>
then in reducer we can count the values HENCE we are retuning .
Ex :-
if we are counting the number of items per sale then we can return
and in reducer use count. A reducer will be called one
for a key each time
Time series aggregation
Key can be complex not necc single string or int or date.
emit ( [year,month,day,location] , dollarAmount )
then in reduce we can use _SUM ( again is applied to Value that is emitted )
u can have keys like [location,yr,month,day] from most significant to least.
this way we can STRUCTURE.
you can try and NOT to write your own reducer , most of the times you can use std reducer.
We can group by loc only or year only ... there is group_level its a number between 0 and length of the array.
[AZ,2010,6,29],s1
[AZ,2010,6,30],s2
if i say group level 3 THEN i get info for [AZ,2010,6]
Note output of Map & reduce is simple text hence keeping comm fast.
Also note all this happening in Real time you don't have a system THAT gets Data from one DB to another overnight.
Parts of a Map Reduce Job
Job Conf
This represents a Job that will be completed via a Map Reduce paradigm
JobConf conf = new
JobConf(MapReduceIntro.class);
Now that you have a JobConfig object, conf, you need to set
the required parameters for the job. These include the input and output
directory locations, the format of the input and output, and the mapper and
reducer classes.
It is good practice to pass in a class that is contained in
the JAR file that has your map and reduce functions.This ensures that
the framework will make the JAR available to the map and reduce tasks run for
your job.
Mapper
Maps
are the individual tasks which transform input records into a intermediate
records. The transformed intermediate records need not be of the same type as
the input records. A given input pair may map to zero or many output pairs.
Configuring the Reduce Phase
To configure the reduce phase, the user must supply the
framework with five pieces of information:
• The number of reduce tasks; if zero, no reduce phase is
run
• The class supplying the reduce method
• The input key and value types for the reduce task; by
default, the same as the reduce output
• The output key and value types for the reduce task
• The output file type for the reduce task output
