com.emc.object.s3

Class LargeFileDownloader

java.lang.Object

com.emc.object.s3.LargeFileDownloader

All Implemented Interfaces:

ProgressListener, java.lang.Runnable

public class LargeFileDownloader
extends java.lang.Object
implements java.lang.Runnable, ProgressListener

Convenience class to facilitate multi-threaded download for large objects. This class will split the object and download it in parts, transferring several parts simultaneously to maximize efficiency.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	LargeFileDownloader.DownloadPartTask

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_PARALLEL_THRESHOLD
static int	DEFAULT_PART_SIZE
static int	DEFAULT_THREADS
static int	MIN_PART_SIZE

Constructor Summary

Constructors

Constructor and Description
LargeFileDownloader(S3Client s3Client, java.lang.String bucket, java.lang.String key, java.io.File file) Creates a new LargeFileDownloader instance that will use s3Client to download bucket/key to file.

Method Summary

Modifier and Type	Method and Description
protected void	doParallelDownload()
protected void	doSingleDownload()
void	download() This method will automatically choose between parallel and single-GET operations based on a configured threshold.
java.lang.String	getBucket()
long	getBytesTransferred()
java.util.concurrent.ExecutorService	getExecutorService()
java.io.File	getFile()
java.lang.String	getKey()
java.lang.Long	getObjectSize()
long	getParallelThreshold()
long	getPartSize()
ProgressListener	getProgressListener()
S3Client	getS3Client()
int	getThreads()
void	progress(long completed, long total) Provides feedback on the number of bytes completed and the total number of bytes to transfer.
void	run()
void	setExecutorService(java.util.concurrent.ExecutorService executorService) Allows for providing a custom thread executor (i.e.
void	setParallelThreshold(long parallelThreshold) Sets the threshold above which parallel operations are used to download, and below which a single-GET is used.
void	setPartSize(long partSize) Sets the size of each part to download.
void	setProgressListener(ProgressListener progressListener)
void	setThreads(int threads) Sets the number of threads to use for transferring parts.
void	transferred(long size) Reports that some bytes have been transferred.
LargeFileDownloader	withExecutorService(java.util.concurrent.ExecutorService executorService)
LargeFileDownloader	withParallelThreshold(long parallelThreshold)
LargeFileDownloader	withPartSize(long partSize)
LargeFileDownloader	withProgressListener(ProgressListener progressListener)
LargeFileDownloader	withThreads(int threads)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PARALLEL_THRESHOLD

public static final int DEFAULT_PARALLEL_THRESHOLD

See Also:

Constant Field Values

MIN_PART_SIZE

public static final int MIN_PART_SIZE

See Also:

Constant Field Values

DEFAULT_PART_SIZE

public static final int DEFAULT_PART_SIZE

See Also:

Constant Field Values

DEFAULT_THREADS

public static final int DEFAULT_THREADS

See Also:

Constant Field Values

Constructor Detail

LargeFileDownloader

public LargeFileDownloader(S3Client s3Client,
                           java.lang.String bucket,
                           java.lang.String key,
                           java.io.File file)

Creates a new LargeFileDownloader instance that will use s3Client to download bucket/key to file.

Method Detail

progress

public void progress(long completed,
                     long total)

Description copied from interface: ProgressListener

Provides feedback on the number of bytes completed and the total number of bytes to transfer.

Specified by:

progress in interface ProgressListener

Parameters:

completed - bytes completely transferred

total - total number of bytes to transfer

transferred

public void transferred(long size)

Description copied from interface: ProgressListener

Reports that some bytes have been transferred. This is a raw method that will be called frequently and can be used for computing current transfer rate. Note that if data is retried, the sum of this method's events may be more than the total object size. For reporting on percent complete, use the progress method instead.

Specified by:

transferred in interface ProgressListener

Parameters:

size - number of bytes transferred

run

public void run()

Specified by:

run in interface java.lang.Runnable

download

public void download()

This method will automatically choose between parallel and single-GET operations based on a configured threshold. Note the default threshold is DEFAULT_PARALLEL_THRESHOLD. Also note that the defaults in this class are optimized for high-speed LAN connectivity. When operating over a WAN or a slower connection, you should reduce the setParallelThreshold(long) parallel threshold} and part size proportionately.

doSingleDownload

protected void doSingleDownload()
                         throws java.io.IOException

Throws:

java.io.IOException

doParallelDownload

protected void doParallelDownload()
                           throws java.lang.Exception

Throws:

java.lang.Exception

getS3Client

public S3Client getS3Client()

getBucket

public java.lang.String getBucket()

getKey

public java.lang.String getKey()

getFile

public java.io.File getFile()

getObjectSize

public java.lang.Long getObjectSize()

getBytesTransferred

public long getBytesTransferred()

getParallelThreshold

public long getParallelThreshold()

setParallelThreshold

public void setParallelThreshold(long parallelThreshold)

Sets the threshold above which parallel operations are used to download, and below which a single-GET is used. This only applies when using the download() method. Note the default threshold is DEFAULT_PARALLEL_THRESHOLD

getPartSize

public long getPartSize()

setPartSize

public void setPartSize(long partSize)

Sets the size of each part to download. Note that 1MB is the minimum part size and the default is DEFAULT_PART_SIZE.

getThreads

public int getThreads()

setThreads

public void setThreads(int threads)

Sets the number of threads to use for transferring parts. thread parts will be transferred in parallel. Default is 6

getExecutorService

public java.util.concurrent.ExecutorService getExecutorService()

setExecutorService

public void setExecutorService(java.util.concurrent.ExecutorService executorService)

Allows for providing a custom thread executor (i.e. for custom thread factories). Note that if you set a custom executor service, the threads property will be ignored.

getProgressListener

public ProgressListener getProgressListener()

setProgressListener

public void setProgressListener(ProgressListener progressListener)

withParallelThreshold

public LargeFileDownloader withParallelThreshold(long parallelThreshold)

withPartSize

public LargeFileDownloader withPartSize(long partSize)

withThreads

public LargeFileDownloader withThreads(int threads)

withExecutorService

public LargeFileDownloader withExecutorService(java.util.concurrent.ExecutorService executorService)

withProgressListener

public LargeFileDownloader withProgressListener(ProgressListener progressListener)