Commands to Run MapReduce Job

ant build your mapreduce job jar first. See the following ant build.xml

<project name=”MyProject” basedir=”.” default=”distr”>

<property name=”Name” value=”TSMapReduce”/>

<property name=”name” value=”TSMapReduce”/>

 

<property name=”version” value=”0.1.0″/>

<property name=”final.name” value=”${name}-${version}”/>

 

<property name=”src.dir” value=”src”/>

    <property name=”lib.dir” value=”lib”/>

 

<property name=”build.dir” value=”build”/>

<property name=”jar.dir” value=”${build.dir}”/>

<property name=”build.lib.dir” value=”${build.dir}/lib”/>

 

<property name=”main-class” value=”com.your_xxx_domain.timeseries.MyMapReduceJob”/>

<property name=”tmp.dir” value=”tmp”/>

    <path id=”classpath”>

        <fileset dir=”${lib.dir}” includes=”**/*.jar”/>

    </path>

 

<target name=”clean”>

<delete dir=”${build.dir}”/>

<delete dir=”${tmp.dir}”/>

</target>

 

<target name=”compile”>

<mkdir dir=”${tmp.dir}”/>

<mkdir dir=”${build.dir}”/>

 

<javac srcdir=”${src.dir}” destdir=”${tmp.dir}” debug=”on”

classpathref=”classpath”/>

</target>

 

<target name=”distr” depends=”jar”>

<mkdir dir=”${build.lib.dir}”/>

<copy todir=”${build.lib.dir}”>

<fileset dir=”${lib.dir}”/>

</copy>

 

<delete dir=”${tmp.dir}”/>

</target>

 

<target name=”jar” depends=”clean, compile”>

 

<jar destfile=”${build.dir}/${final.name}.jar” basedir=”${tmp.dir}”>

<!– <manifest>

<attribute name=”Main-Class” value=”${main-class}”/>

<attribute name=”Class-Path” value=”classpath”/>

</manifest>

–>

<fileset dir=”.”>

<include name=”**/${lib.dir}/**” />

</fileset>

 

</jar>

</target>

</project>

./bin/hadoop jar YOUR_MAP_REDUCE_JOB.jar com.your_xxx_domain.path.to.your.class.MyMapReduceClass programArgs1 programArgs2

Mahout distance measure

All the distance measure classes are in org.apache.mahout.common.distance package.

They all implements the DistanceMeasure interface which also extends the Parametered interface.

There are two methods in the DistanceMeasure interface:

double distance(Vector v1, Vector v2)

double distance(double centroidLengthSquare, Vector centroid, Vector v)

You can create a new distance measure by implementing the DistanceMeasure interface 🙂

Mahout clustering

I have been exploring the mahout clustering packages. Some brief description of the codes

org.apache.mahout.clustering.kmeans.KMeansDriver is the main entry point. The run method setup the clustering job by specifying the input path to data points, path to initial k clusters, distance measure class used.

In the KMeansDriver, it also setup the settings in the Configuration as follows:

conf.set(KMeansConfigKeys.CLUSTER_PATH_KEY, clustersIn.toString());
conf.set(KMeansConfigKeys.DISTANCE_MEASURE_KEY, measureClass);
conf.set(KMeansConfigKeys.CLUSTER_CONVERGENCE_KEY, convergenceDelta);

KMeansConfigKeys is an interface which defines the config keys for the settings.

org.apache.mahout.clustering.kmeans.KMeansMapper loads the initial predefined clusters via the setup method. In the map method, it reads the VectorWritable and passes it to the KMeansCluster to do the clustering, comparing against the predefined clusters via the following call.

this.clusterer.emitPointToNearestCluster(point.get(), this.clusters, context);

In the emitPointToNeareastCluster, it finds the nearest cluster by comparing the distance to each cluster in the list. It then writes the cluster identifier and wraps the point in the ClusterObservations to HDFS.

context.write(new Text(nearestCluster.getIdentifier()), new ClusterObservations(1, point, point.times(point)));

In the KMeansReducer, it aggregates the list of ClusterObservations into a Cluster before writing it the Cluster id as key and Cluster as the value.

In the setup method, KMeansReducer loads the predefined initial cluster from HDFS path and creates the ClusterMap of key as cluster id and value as Cluster itself.

Different file formats used in Hadoop and HBase

I have been investigating different file formats used in Hadoop and HBase to understand how these file formats assist in the speedup that we’ve all witnessed in this Hadoop big data world. Also, I recommend all java developer to dig into Hadoop and HBase source code because you will definitely learn a lot and improve your java skills.

File formats used in Hadoop are SequenceFile, TFile, and Avro file whereas HFile is used exclusively in HBase.

I found an interesting and detailed explanation of the internal structure of HFile representation in the following blog.

http://cloudepr.blogspot.com/2009/09/hfile-block-indexed-file-format-to.html

Enjoy

BigDataExplorer