org.apache.hadoop.mapred
Class JobClient

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.mapred.JobClient

All Implemented Interfaces:

Configurable, Tool

public class JobClient
extends Configured
implements Tool

JobClient is the primary interface for the user-job to interact with the JobTracker. JobClient provides facilities to submit jobs, track their progress, access component-tasks' reports/logs, get the Map-Reduce cluster status information etc.

The job submission process involves:

1. Checking the input and output specifications of the job.
2. Computing the InputSplits for the job.
3. Setup the requisite accounting information for the DistributedCache of the job, if necessary.
4. Copying the job's jar and configuration to the map-reduce system directory on the distributed file-system.
5. Submitting the job to the JobTracker and optionally monitoring it's status.

Normally the user creates the application, describes various facets of the job via JobConf and then uses the JobClient to submit the job and monitor its progress.

Here is an example on how to use JobClient:

     // Create a new JobConf
     JobConf job = new JobConf(new Configuration(), MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     JobClient.runJob(job);
 

Job Control

At times clients would chain map-reduce jobs to accomplish complex tasks which cannot be done via a single map-reduce job. This is fairly easy since the output of the job, typically, goes to distributed file-system and that can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such situations the various job-control options are:

1. runJob(JobConf) : submits the job and returns only after the job has completed.
2. submitJob(JobConf) : only submits the job, then poll the returned handle to the RunningJob to query status and make scheduling decisions.
3. JobConf.setJobEndNotificationURI(String) : setup a notification on job-completion, thus avoiding polling.

See Also:

JobConf, ClusterStatus, Tool, DistributedCache

Nested Class Summary

static class

JobClient.TaskStatusFilter

Field Summary

static int	CLUSTER_INCREMENT
static long	COUNTER_UPDATE_INTERVAL
static int	FILE_NOT_FOUND
static int	HEARTBEAT_INTERVAL_MIN
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static float	MAX_INMEM_FILESIZE_FRACTION Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem.
static float	MAX_INMEM_FILESYS_USE Constant denoting when a merge of in memory files will be triggered
static String	RAW_MAP_OUTPUT_LENGTH The custom http header used for the "raw" map output length.
static int	SUCCESS
static String	TEMP_DIR_NAME Temporary directory name
static String	WORKDIR

Constructor Summary

JobClient()
Create a job client.

JobClient(InetSocketAddress jobTrackAddr, Configuration conf)
Build a job client, connect to the indicated job tracker.

JobClient(JobConf conf)
Build a job client with the given JobConf, and connect to the default JobTracker.

Method Summary

void	close() Close the JobClient.
JobStatus[]	getAllJobs() Get the jobs that are submitted.
ClusterStatus	getClusterStatus() Get status information about the Map-Reduce cluster.
static Configuration	getCommandLineConfig() return the command line configuration
int	getDefaultMaps() Get status information about the max available Maps in the cluster.
int	getDefaultReduces() Get status information about the max available Reduces in the cluster.
FileSystem	getFs() Get a filesystem handle.
RunningJob	getJob(JobID jobid) Get an RunningJob object to track an ongoing job.
RunningJob	getJob(String jobid) Deprecated. Applications should rather use getJob(JobID).
TaskReport[]	getMapTaskReports(JobID jobId) Get the information of the current state of the map tasks of a job.
TaskReport[]	getMapTaskReports(String jobId) Deprecated. Applications should rather use getMapTaskReports(JobID)
TaskReport[]	getReduceTaskReports(JobID jobId) Get the information of the current state of the reduce tasks of a job.
TaskReport[]	getReduceTaskReports(String jobId) Deprecated. Applications should rather use getReduceTaskReports(JobID)
Path	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
JobClient.TaskStatusFilter	getTaskOutputFilter() Deprecated.
static JobClient.TaskStatusFilter	getTaskOutputFilter(JobConf job) Get the task output filter out of the JobConf.
void	init(JobConf conf) Connect to the default JobTracker.
JobStatus[]	jobsToComplete() Get the jobs that are not completed and not failed.
static void	main(String[] argv)
int	run(String[] argv) Execute the command with the given arguments.
static RunningJob	runJob(JobConf job) Utility that submits a job, then polls for progress until the job is complete.
void	setTaskOutputFilter(JobClient.TaskStatusFilter newValue) Deprecated.
static void	setTaskOutputFilter(JobConf job, JobClient.TaskStatusFilter newValue) Modify the JobConf to set the task output filter.
RunningJob	submitJob(JobConf job) Submit a job to the MR system.
RunningJob	submitJob(String jobFile) Submit a job to the MR system.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN

See Also:

Constant Field Values

CLUSTER_INCREMENT

public static final int CLUSTER_INCREMENT

See Also:

Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL

See Also:

Constant Field Values

MAX_INMEM_FILESYS_USE

public static final float MAX_INMEM_FILESYS_USE

Constant denoting when a merge of in memory files will be triggered

See Also:

Constant Field Values

MAX_INMEM_FILESIZE_FRACTION

public static final float MAX_INMEM_FILESIZE_FRACTION

Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem. Ideally, this should be a factor of MAX_INMEM_FILESYS_USE

See Also:

Constant Field Values

SUCCESS

public static final int SUCCESS

See Also:

Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

See Also:

Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

See Also:

Constant Field Values

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH

The custom http header used for the "raw" map output length.

See Also:

Constant Field Values

TEMP_DIR_NAME

public static final String TEMP_DIR_NAME

Temporary directory name

See Also:

Constant Field Values

WORKDIR

public static final String WORKDIR

See Also:

Constant Field Values

Constructor Detail

JobClient

public JobClient()

Create a job client.

JobClient

public JobClient(JobConf conf)
          throws IOException

Build a job client with the given JobConf, and connect to the default JobTracker.

Parameters:

conf - the job configuration.

Throws:

IOException

JobClient

public JobClient(InetSocketAddress jobTrackAddr,
                 Configuration conf)
          throws IOException

Build a job client, connect to the indicated job tracker.

Parameters:

jobTrackAddr - the job tracker to connect to.

conf - configuration.

Throws:

IOException

Method Detail

getCommandLineConfig

public static Configuration getCommandLineConfig()

return the command line configuration

init

public void init(JobConf conf)
          throws IOException

Connect to the default JobTracker.

Parameters:

conf - the job configuration.

Throws:

IOException

close

public void close()
           throws IOException

Close the JobClient.

Throws:

IOException

getFs

public FileSystem getFs()
                 throws IOException

Get a filesystem handle. We need this to prepare jobs for submission to the MapReduce system.

Returns:

the filesystem handle.

Throws:

IOException

submitJob

public RunningJob submitJob(String jobFile)
                     throws FileNotFoundException,
                            InvalidJobConfException,
                            IOException

Submit a job to the MR system. This returns a handle to the RunningJob which can be used to track the running-job.

Parameters:

jobFile - the job configuration.

Returns:

a handle to the RunningJob which can be used to track the running-job.

Throws:

FileNotFoundException

InvalidJobConfException

IOException

submitJob

public RunningJob submitJob(JobConf job)
                     throws FileNotFoundException,
                            InvalidJobConfException,
                            IOException

Submit a job to the MR system. This returns a handle to the RunningJob which can be used to track the running-job.

Parameters:

job - the job configuration.

Returns:

a handle to the RunningJob which can be used to track the running-job.

Throws:

FileNotFoundException

InvalidJobConfException

IOException

getJob

public RunningJob getJob(JobID jobid)
                  throws IOException

Get an RunningJob object to track an ongoing job. Returns null if the id does not correspond to any known job.

Parameters:

jobid - the jobid of the job.

Returns:

the RunningJob handle to track the job, null if the jobid doesn't correspond to any known job.

Throws:

IOException

getJob

@Deprecated
public RunningJob getJob(String jobid)
                  throws IOException

Deprecated. Applications should rather use getJob(JobID).

Throws:

IOException

getMapTaskReports

public TaskReport[] getMapTaskReports(JobID jobId)
                               throws IOException

Get the information of the current state of the map tasks of a job.

Parameters:

jobId - the job to query.

Returns:

the list of all of the map tips.

Throws:

IOException

getMapTaskReports

@Deprecated
public TaskReport[] getMapTaskReports(String jobId)
                               throws IOException

Deprecated. Applications should rather use getMapTaskReports(JobID)

Throws:

IOException

getReduceTaskReports

public TaskReport[] getReduceTaskReports(JobID jobId)
                                  throws IOException

Get the information of the current state of the reduce tasks of a job.

Parameters:

jobId - the job to query.

Returns:

the list of all of the reduce tips.

Throws:

IOException

getReduceTaskReports

@Deprecated
public TaskReport[] getReduceTaskReports(String jobId)
                                  throws IOException

Deprecated. Applications should rather use getReduceTaskReports(JobID)

Throws:

IOException

getClusterStatus

public ClusterStatus getClusterStatus()
                               throws IOException

Get status information about the Map-Reduce cluster.

Returns:

the status information about the Map-Reduce cluster as an object of ClusterStatus.

Throws:

IOException

jobsToComplete

public JobStatus[] jobsToComplete()
                           throws IOException

Get the jobs that are not completed and not failed.

Returns:

array of JobStatus for the running/to-be-run jobs.

Throws:

IOException

getAllJobs

public JobStatus[] getAllJobs()
                       throws IOException

Get the jobs that are submitted.

Returns:

array of JobStatus for the submitted jobs.

Throws:

IOException

runJob

public static RunningJob runJob(JobConf job)
                         throws IOException

Utility that submits a job, then polls for progress until the job is complete.

Parameters:

job - the job configuration.

Throws:

IOException

setTaskOutputFilter

@Deprecated
public void setTaskOutputFilter(JobClient.TaskStatusFilter newValue)

Deprecated.

Sets the output filter for tasks. only those tasks are printed whose output matches the filter.

Parameters:

newValue - task filter.

getTaskOutputFilter

public static JobClient.TaskStatusFilter getTaskOutputFilter(JobConf job)

Get the task output filter out of the JobConf.

Parameters:

job - the JobConf to examine.

Returns:

the filter level.

setTaskOutputFilter

public static void setTaskOutputFilter(JobConf job,
                                       JobClient.TaskStatusFilter newValue)

Modify the JobConf to set the task output filter.

Parameters:

job - the JobConf to modify.

newValue - the value to set.

getTaskOutputFilter

@Deprecated
public JobClient.TaskStatusFilter getTaskOutputFilter()

Deprecated.

Returns task output filter.

Returns:

task filter.

run

public int run(String[] argv)
        throws Exception

Description copied from interface: Tool

Execute the command with the given arguments.

Specified by:

run in interface Tool

Parameters:

argv - command specific arguments.

Returns:

exit code.

Throws:

Exception

getDefaultMaps

public int getDefaultMaps()
                   throws IOException

Get status information about the max available Maps in the cluster.

Returns:

the max available Maps in the cluster

Throws:

IOException

getDefaultReduces

public int getDefaultReduces()
                      throws IOException

Get status information about the max available Reduces in the cluster.

Returns:

the max available Reduces in the cluster

Throws:

IOException

getSystemDir

public Path getSystemDir()

Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:

the system directory where job-specific files are to be placed.

main

public static void main(String[] argv)
                 throws Exception

Throws:

Exception