org.apache.hadoop.dfs
Class DataNode

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.dfs.DataNode

All Implemented Interfaces:

Runnable, Configurable, FSConstants

public class DataNode
extends Configured
implements FSConstants, Runnable

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.dfs.FSConstants

FSConstants.CheckpointStates, FSConstants.DatanodeReportType, FSConstants.NodeType, FSConstants.SafeModeAction, FSConstants.StartupOption, FSConstants.UpgradeAction

Field Summary

static org.apache.commons.logging.Log	LOG
static long	versionID 3: added a finalize parameter to updateBlock
static long	versionID 1: start of time

Fields inherited from interface org.apache.hadoop.dfs.FSConstants

BLOCK_INVALIDATE_CHUNK, BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, COMPLETE_SUCCESS, DATA_TRANSFER_VERSION, DEFAULT_BLOCK_SIZE, DEFAULT_DATA_SOCKET_SIZE, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, OP_ACK, OP_BLOCKRECEIVED, OP_BLOCKREPORT, OP_CLIENT_ABANDONBLOCK, OP_CLIENT_ABANDONBLOCK_ACK, OP_CLIENT_ADDBLOCK, OP_CLIENT_ADDBLOCK_ACK, OP_CLIENT_COMPLETEFILE, OP_CLIENT_COMPLETEFILE_ACK, OP_CLIENT_DATANODE_HINTS, OP_CLIENT_DATANODE_HINTS_ACK, OP_CLIENT_DATANODEREPORT, OP_CLIENT_DATANODEREPORT_ACK, OP_CLIENT_DELETE, OP_CLIENT_DELETE_ACK, OP_CLIENT_EXISTS, OP_CLIENT_EXISTS_ACK, OP_CLIENT_ISDIR, OP_CLIENT_ISDIR_ACK, OP_CLIENT_LISTING, OP_CLIENT_LISTING_ACK, OP_CLIENT_MKDIRS, OP_CLIENT_MKDIRS_ACK, OP_CLIENT_OBTAINLOCK, OP_CLIENT_OBTAINLOCK_ACK, OP_CLIENT_OPEN, OP_CLIENT_OPEN_ACK, OP_CLIENT_RAWSTATS, OP_CLIENT_RAWSTATS_ACK, OP_CLIENT_RELEASELOCK, OP_CLIENT_RELEASELOCK_ACK, OP_CLIENT_RENAMETO, OP_CLIENT_RENAMETO_ACK, OP_CLIENT_RENEW_LEASE, OP_CLIENT_RENEW_LEASE_ACK, OP_CLIENT_STARTFILE, OP_CLIENT_STARTFILE_ACK, OP_CLIENT_TRYAGAIN, OP_COPY_BLOCK, OP_ERROR, OP_FAILURE, OP_HEARTBEAT, OP_INVALIDATE_BLOCKS, OP_READ_BLOCK, OP_READ_METADATA, OP_REPLACE_BLOCK, OP_STATUS_CHECKSUM_OK, OP_STATUS_ERROR, OP_STATUS_ERROR_CHECKSUM, OP_STATUS_ERROR_EXISTS, OP_STATUS_ERROR_INVALID, OP_STATUS_SUCCESS, OP_TRANSFERBLOCKS, OP_TRANSFERDATA, OP_WRITE_BLOCK, OPERATION_FAILED, READ_TIMEOUT, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE, STILL_WAITING, WRITE_TIMEOUT, WRITE_TIMEOUT_EXTENSION

Method Summary

static InetSocketAddress	createSocketAddr(String target) Deprecated.
org.apache.hadoop.dfs.BlockMetaDataInfo	getBlockMetaDataInfo(org.apache.hadoop.dfs.Block block)
static DataNode	getDataNode() Return the DataNode object
FSDatasetInterface	getFSDataset() This method is used for testing.
String	getNamenode() Return the namenode's identifier
InetSocketAddress	getNameNodeAddr()
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
InetSocketAddress	getSelfAddr()
static void	main(String[] args)
void	offerService() Main loop for the DataNode.
org.apache.hadoop.dfs.Block	recoverBlock(org.apache.hadoop.dfs.Block block, DatanodeInfo[] targets) Start generation-stamp recovery for specified block
void	run() No matter what kind of exception we get, keep retrying to offerService().
void	scheduleBlockReport(long delay) This methods arranges for the data node to send the block report at the next heartbeat.
void	shutdown() Shut down this instance of the datanode.
String	toString()
void	updateBlock(org.apache.hadoop.dfs.Block oldblock, org.apache.hadoop.dfs.Block newblock, boolean finalize) Update the block to the new generation stamp and length.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

versionID

public static final long versionID

3: added a finalize parameter to updateBlock

See Also:

Constant Field Values

versionID

public static final long versionID

1: start of time

See Also:

Constant Field Values

Method Detail

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)
                                          throws IOException

Deprecated.

Use NetUtils.createSocketAddr(String) instead.

Throws:

IOException

getDataNode

public static DataNode getDataNode()

Return the DataNode object

getNameNodeAddr

public InetSocketAddress getNameNodeAddr()

getSelfAddr

public InetSocketAddress getSelfAddr()

getNamenode

public String getNamenode()

Return the namenode's identifier

shutdown

public void shutdown()

Shut down this instance of the datanode. Returns only after shutdown is complete.

offerService

public void offerService()
                  throws Exception

Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

Throws:

Exception

run

public void run()

No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

Specified by:

run in interface Runnable

toString

public String toString()

Overrides:

toString in class Object

scheduleBlockReport

public void scheduleBlockReport(long delay)

This methods arranges for the data node to send the block report at the next heartbeat.

getFSDataset

public FSDatasetInterface getFSDataset()

This method is used for testing. Examples are adding and deleting blocks directly. The most common usage will be when the data node's storage is similated.

Returns:

the fsdataset that stores the blocks

main

public static void main(String[] args)

getBlockMetaDataInfo

public org.apache.hadoop.dfs.BlockMetaDataInfo getBlockMetaDataInfo(org.apache.hadoop.dfs.Block block)
                                                             throws IOException

Returns:

the BlockMetaDataInfo of a block; null if the block is not found

Throws:

IOException

updateBlock

public void updateBlock(org.apache.hadoop.dfs.Block oldblock,
                        org.apache.hadoop.dfs.Block newblock,
                        boolean finalize)
                 throws IOException

Update the block to the new generation stamp and length.

Throws:

IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Return protocol version corresponding to protocol interface.

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException

recoverBlock

public org.apache.hadoop.dfs.Block recoverBlock(org.apache.hadoop.dfs.Block block,
                                                DatanodeInfo[] targets)
                                         throws IOException

Start generation-stamp recovery for specified block

Parameters:

block - the specified block

Returns:

the new blockid if recovery successful and the generation stamp got updated as part of the recovery, else returns null if the block id not have any data and the block was deleted.

Throws:

IOException