org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator

Using this class, we can iterate through the values in the sequence file. Key is expected to be of Writable.

Class<? extends Writable> keyClass = (Class<? extends Writable>) reader.getKeyClass();

All the distance measure classes are in org.apache.mahout.common.distance package.

They all implements the DistanceMeasure interface which also extends the Parametered interface.

There are two methods in the DistanceMeasure interface:

double distance(Vector v1, Vector v2)

double distance(double centroidLengthSquare, Vector centroid, Vector v)

You can create a new distance measure by implementing the DistanceMeasure interface 🙂

I have been exploring the mahout clustering packages. Some brief description of the codes

org.apache.mahout.clustering.kmeans.KMeansDriver is the main entry point. The run method setup the clustering job by specifying the input path to data points, path to initial k clusters, distance measure class used.

In the KMeansDriver, it also setup the settings in the Configuration as follows:

conf.set(KMeansConfigKeys.CLUSTER_PATH_KEY, clustersIn.toString());
conf.set(KMeansConfigKeys.DISTANCE_MEASURE_KEY, measureClass);
conf.set(KMeansConfigKeys.CLUSTER_CONVERGENCE_KEY, convergenceDelta);

KMeansConfigKeys is an interface which defines the config keys for the settings.

org.apache.mahout.clustering.kmeans.KMeansMapper loads the initial predefined clusters via the setup method. In the map method, it reads the VectorWritable and passes it to the KMeansCluster to do the clustering, comparing against the predefined clusters via the following call.

this.clusterer.emitPointToNearestCluster(point.get(), this.clusters, context);

In the emitPointToNeareastCluster, it finds the nearest cluster by comparing the distance to each cluster in the list. It then writes the cluster identifier and wraps the point in the ClusterObservations to HDFS.

context.write(new Text(nearestCluster.getIdentifier()), new ClusterObservations(1, point, point.times(point)));

In the KMeansReducer, it aggregates the list of ClusterObservations into a Cluster before writing it the Cluster id as key and Cluster as the value.

In the setup method, KMeansReducer loads the predefined initial cluster from HDFS path and creates the ClusterMap of key as cluster id and value as Cluster itself.