Enums why cant we override equals

in the java.lang.Enum class equals method is like

 
 public final boolean equals(Object other) {
       return this==other;
 }
 
actually overriding it with something else does not make sense hence its final. As for Enums equals() and == are the same things.

Java Generic < ? > vs < T >

Using "?" is the same as "any", whereas "T" means "a specific type". So, compare these interfaces:

public interface StrictClass {
 public T doFunction(Class class);
}


public interface EasyClass {
 public < ? >  doFunction(Class class);
}


When to use which one

There are also use cases for choosing over (or vice versa) that apply when you don't add type parameter to the class that encloses the method. For example, consider the difference between

public boolean add(List j) {
    boolean t = true;
    for (JLabel b : j) {
        if (b instanceof JLabel) {
            t = t && labels.add(b);
        }
    }
    return t;
}
and

public boolean add(List j) {
    boolean t = true;
    for (JLabel b : j) {
        if (b instanceof JLabel) {
            t = t && labels.add(b);
        }
    }
    return t;
}
The first method will actually not compile UNLESS you add an appropriate type parameter to the enclosing class, whereas the second method WILL compile regardless of whether the enclosing class has a type parameter

Using Type Tokens to Retrieve Generic Parameters

Interesting read

http://www.jquantlib.org/index.php/Using_TypeTokens_to_retrieve_generic_parameters

Type Erasure : Why the information is dropped at compile time

We all know that, at compile-time, the compiler has full type information available but this information is intentionally dropped in general when the binary code is generated, in a process known as type erasure. 

So to put in different words this feature is offered by javac :)

one needs to realize the concept of type erasure derives from a need of compatibility with previous versions of java.

• Source compatibility (Nice to have...)
• Binary compatibility (Must have!)
• Migration compatibility
  
  • Existing programs must continue to work
  • Existing libraries must be able to use generic types
  • Must have!

This is done this way due to compatibility issues... The intention of language designers was providing full source code compatibility and full binary code compatibility between versions of the platform. If it was implemented differently, you would have to recompile your legacy applications when you migrate to newer versions of the platform. The way it was done, all method signatures are preserved (source code compatibility) and you don't need to recompile anything (binary compatibility).

Hibernate Query vs Criteria Performance

For some reason HQL seems more faster than Criteria

If we write a query like

select count(*) from R r where r.ISREPLACEDBY = 0 and r.STATUS='OK' and r.A = ? and r.C in (select distinct RC from CX cx where cx.FROMDATE >= ? and cx.FROMDATE <= ?)

Using both HQL and Criteria , then HQL will run much faster.

It seems that the criteria api creates new variable names each time a prepared statement is executed. The database (in our case, DB2) calculates then a new query execution plan each time the statement is executed. On the other hand, HQL uses the same variable names, allowing the database to re-use the query execution plans.

Another issue i noticed when using HQL , I was using HQL like

from employee emp where emp.id=123
at one place, and HQL like
from employee e where e.id=123
at another place.

Although i was using a query level cache BUT still it was not being cached BECAUSE query cache was not being used here as the key for query cache is the query itself.

Hadoop : Was the job really successful

An accurate determination of success is critical. 

The check for success primarily involves ensuring that the number of records output is roughly the same as the number of records input. Hadoop jobs are generally dealing with bulk real world data, which is never 100% clean, so a small error rate is generally acceptable.

It is a good practice to wrap your map and reduce methods in a try block that catches Throwables and reports on the catches.

Each call on the reporter object or the output collector provides a heartbeat to the framework,

                reporter.incrCounter( "Input", "total records", 1 );

                reporter.incrCounter( "Input", "parsed records", 1 );

                reporter.incrCounter( "Input", "number format", 1 );      

                reporter.incrCounter( "Input", "Exception", 1 );

                // better to use ENUMS to avoid spelling mistakes or extra spaces in end

if (format != 0) {

logger.warn( "There were " + format + " keys that were not "+ "transformable to long values");

}

/** Check to see if we had any unexpected exceptions. This usually indicates some significant problem, either with the machine running the task that had the exception, or the map or reduce function code. Log an error for each type of exception with the count.

*/

if (exceptions > 0 ) {

                Counters.Group exceptionGroup = jobCounters.getGroup(

                TransformKeysToLongMapper.EXCEPTIONS );

                for (Counters.Counter counter : exceptionGroup) {

                                logger.error( "There were " + counter.getCounter()

                                + " exceptions of type " + counter.getDisplayName() );

                }

}

if (total == parsed) {

                logger.info("The job completed successfully.");

                System.exit(0);

}

// We had some failures in handling the input records. Did enough records process for this to be a

// successful job is 90% good enough?

if (total * .9 <= parsed) {

logger.warn( "The job completed with some errors, "+ (total - parsed) + " out of " + total );

System.exit( 0 );

}

logger.error( "The job did not complete successfully,"+" too many errors processing the input, only "

+ parsed + " of " + total + "records completed" );

System.exit( 1 );

Hadoop : Map Reduce

Its old design pattern which is more relevant now. You can have different parts of calculation run on different machines which can talk to each other and hence run calculation faster. 

Google have their own implementation which is similar to Cloudera implementation. Then Amazon has Hadoop Lite , houch DB or mongo DB have some implementation running inside. Each one is solving specific probs.

MapReduce is basically made up of MAP and REDUCE. Where MAP is Stateless and independent and REDUCE for it we need to think of how it fits into overall flow... so there is seperation. Let's think of sorting prob each node needs to sort its data and fit it into Reducer.

Let's suppose a cluster of N nodes having data and running a piece of code F(x) (like sum , avg etc) . Now there is no separation of data ex it's not the case one node has Sept data other Oct node. not data is separated by Product type etc , data is evenly distributed. AND DATA is independent.

Each of the nodes will output pairs. can be date and dollar amount(sales).

Seems Can have more than one reducer. All the key value pairs come to this Reducer and gives you any key value pair ie sales on any given day.

Can do similar stuff in SQL , so whats the advtg here ? First this is not running on enterprise architecture think in terms of clustered architecture. And secondly Advtg here is scalability ... independent portions don't have to wait for anyone to finish.

Example :- suppose 6 nodes and we have 2 reducers one for 1st quarter Jan - Mar and another for second march so each reducers gets data from all of the nodes. 

Map is like a Data Warehouse code which extracts , Transform and loads.

It's not necessary to always write/need reduce , 6 out of 10 times you can pass by the reduce code.

Can also chain map reduce.

Icremental Map Reduce :- Couch DB does this once you ask it a ques ( avg sale this quarter ) it keeps computing this for you. this makes it different from google. Couch DB is interested in 1 to 100 DB server and Google is interested in PeraByte plus info like how to scale the web.  COuch DB is a doc based database where docs are represented as JSON when intreacting with it , else stored as binary.

Suppose i want to find biggest sales. so sorting on each node. sort is done according to key so you need to put the right thing in the key. so basically ask yourself how do i want to query this thing later on ... that defines the key.

you can do lot of diff things with Emit(amount,null) if date is of june then emit if date is in last 30 days then emit else not.

Things like count , sum , min , max , avg , std dev etc are written to the DB.

Ex :- we want to know the total no of sales of certain amount. so basically in mapper we return k,v where k should be amount .. in the mapper code we can have logic if greater than x then return k,v => then in reducer we can count the values HENCE we are retuning .

Ex :- if we are counting the number of items per sale then we can return and in reducer use count. A reducer will be called one for a key each time

Time series aggregation 

Key can be complex not necc single string or int or date.

 emit ( [year,month,day,location] , dollarAmount )

then in reduce we can use _SUM ( again is applied to Value that is emitted )

u can have keys like [location,yr,month,day] from most significant to least.

this way we can STRUCTURE.

you can try and NOT to write your own reducer , most of the times you can use std reducer.

We can group by loc only or year only ... there is group_level its a number between 0 and length of the array. 

[AZ,2010,6,29],s1

[AZ,2010,6,30],s2

if i say group level 3 THEN i get info for [AZ,2010,6]

Note output of Map & reduce is simple text hence keeping comm fast.

Also note all this happening in Real time you don't have a system THAT gets Data from one DB to another overnight.

Parts of a Map Reduce Job

Job Conf

This represents a Job that will be completed via a Map Reduce paradigm

                              JobConf conf = new JobConf(MapReduceIntro.class);

Now that you have a JobConfig object, conf, you need to set the required parameters for the job. These include the input and output directory locations, the format of the input and output, and the mapper and reducer classes.

It is good practice to pass in a class that is contained in the JAR file that has your map and reduce functions.This ensures that the framework will make the JAR available to the map and reduce tasks run for your job.

Mapper

 Maps are the individual tasks which transform input records into a intermediate records. The transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or many output pairs.

Configuring the Reduce Phase

To configure the reduce phase, the user must supply the framework with five pieces of information:

• The number of reduce tasks; if zero, no reduce phase is run

• The class supplying the reduce method

• The input key and value types for the reduce task; by default, the same as the reduce output

• The output key and value types for the reduce task

• The output file type for the reduce task output