org.apache.hadoop.dfs
Class ChecksumDistributedFileSystem

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.fs.FileSystem
          org.apache.hadoop.fs.FilterFileSystem
              org.apache.hadoop.fs.ChecksumFileSystem
                  org.apache.hadoop.dfs.ChecksumDistributedFileSystem

All Implemented Interfaces:

Closeable, Configurable

public class ChecksumDistributedFileSystem
extends ChecksumFileSystem

An implementation of ChecksumFileSystem over DistributedFileSystem. Note that as of now (May 07), DistributedFileSystem natively checksums all of its data. Using this class is not be necessary in most cases. Currently provided mainly for backward compatibility and testing.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.Statistics

Field Summary

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs

Fields inherited from class org.apache.hadoop.fs.FileSystem

LOG, statistics

Constructor Summary

ChecksumDistributedFileSystem()

ChecksumDistributedFileSystem(InetSocketAddress namenode, Configuration conf)
Deprecated.

Method Summary

UpgradeStatusReport	distributedUpgradeProgress(FSConstants.UpgradeAction action)
void	finalizeUpgrade() Finalize previously upgraded files system state.
DatanodeInfo[]	getDataNodeStats() Return statistics for each datanode.
FileStatus	getFileStatus(Path f) Returns the stat information about the file.
long	getRawCapacity() Return the total raw capacity of the filesystem, disregarding replication .
long	getRawUsed() Return the total raw used space in the filesystem, disregarding replication .
void	metaSave(String pathname)
void	refreshNodes()
boolean	reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos) We need to find the blocks that didn't match.
boolean	setSafeMode(FSConstants.SafeModeAction action) Enter, leave or get safe mode.

Methods inherited from class org.apache.hadoop.fs.ChecksumFileSystem

completeLocalOutput, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, delete, getApproxChkSumLength, getBytesPerSum, getChecksumFile, getChecksumFileLength, getChecksumLength, getRawFileSystem, isChecksumFile, listStatus, mkdirs, open, rename, setConf, setReplication, startLocalOutput

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

append, checkPath, close, create, delete, getConf, getDefaultBlockSize, getDefaultReplication, getFileBlockLocations, getFileBlockLocations, getHomeDirectory, getName, getUri, getWorkingDirectory, initialize, makeQualified, mkdirs, setOwner, setPermission, setWorkingDirectory

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, closeAll, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, createNewFile, deleteOnExit, exists, get, get, getBlockSize, getContentSummary, getDefaultUri, getLength, getLocal, getNamed, getReplication, getStatistics, getUsed, globStatus, globStatus, isDirectory, isFile, listStatus, listStatus, listStatus, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, open, parseArgs, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ChecksumDistributedFileSystem

public ChecksumDistributedFileSystem()

ChecksumDistributedFileSystem

public ChecksumDistributedFileSystem(InetSocketAddress namenode,
                                     Configuration conf)
                              throws IOException

Deprecated.

Throws:

IOException

Method Detail

getRawCapacity

public long getRawCapacity()
                    throws IOException

Return the total raw capacity of the filesystem, disregarding replication .

Throws:

IOException

getRawUsed

public long getRawUsed()
                throws IOException

Return the total raw used space in the filesystem, disregarding replication .

Throws:

IOException

getDataNodeStats

public DatanodeInfo[] getDataNodeStats()
                                throws IOException

Return statistics for each datanode.

Throws:

IOException

setSafeMode

public boolean setSafeMode(FSConstants.SafeModeAction action)
                    throws IOException

Enter, leave or get safe mode.

Throws:

IOException

See Also:

ClientProtocol.setSafeMode(FSConstants.SafeModeAction)

refreshNodes

public void refreshNodes()
                  throws IOException

Throws:

IOException

finalizeUpgrade

public void finalizeUpgrade()
                     throws IOException

Finalize previously upgraded files system state.

Throws:

IOException

distributedUpgradeProgress

public UpgradeStatusReport distributedUpgradeProgress(FSConstants.UpgradeAction action)
                                               throws IOException

Throws:

IOException

metaSave

public void metaSave(String pathname)
              throws IOException

Throws:

IOException

reportChecksumFailure

public boolean reportChecksumFailure(Path f,
                                     FSDataInputStream in,
                                     long inPos,
                                     FSDataInputStream sums,
                                     long sumsPos)

We need to find the blocks that didn't match. Likely only one is corrupt but we will report both to the namenode. In the future, we can consider figuring out exactly which block is corrupt.

Overrides:

reportChecksumFailure in class ChecksumFileSystem

Parameters:

f - the file name containing the error

in - the stream open on the file

inPos - the position of the beginning of the bad data in the file

sums - the stream open on the checksum file

sumsPos - the position of the beginning of the bad data in the checksum file

Returns:

if retry is neccessary

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Returns the stat information about the file.

Overrides:

getFileStatus in class FilterFileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException