Package com.emc.object.s3

Class LargeFileDownloader

  • All Implemented Interfaces:
    ProgressListener, java.lang.Runnable
    public class LargeFileDownloader
extends java.lang.Object
implements java.lang.Runnable, ProgressListener
    Convenience class to facilitate multi-threaded download for large objects. This class will split the object and download it in parts, transferring several parts simultaneously to maximize efficiency.

    • Constructor Detail

      • LargeFileDownloader

        public LargeFileDownloader​(S3Client s3Client,
                           java.lang.String bucket,
                           java.lang.String key,
                           java.io.File file)
        Creates a new LargeFileDownloader instance that will use s3Client to download bucket/key to file.

    • Method Detail

      • progress

        public void progress​(long completed,
                     long total)
        Description copied from interface: ProgressListener
        Provides feedback on the number of bytes completed and the total number of bytes to transfer.
        Specified by:
        progress in interface ProgressListener
        Parameters:
        completed - bytes completely transferred
        total - total number of bytes to transfer

      • transferred

        public void transferred​(long size)
        Description copied from interface: ProgressListener
        Reports that some bytes have been transferred. This is a raw method that will be called frequently and can be used for computing current transfer rate. Note that if data is retried, the sum of this method's events may be more than the total object size. For reporting on percent complete, use the progress method instead.
        Specified by:
        transferred in interface ProgressListener
        Parameters:
        size - number of bytes transferred

      • run

        public void run()
        Specified by:
        run in interface java.lang.Runnable

      • download

        public void download()
        This method will automatically choose between parallel and single-GET operations based on a configured threshold. Note the default threshold is DEFAULT_PARALLEL_THRESHOLD. Also note that the defaults in this class are optimized for high-speed LAN connectivity. When operating over a WAN or a slower connection, you should reduce the setParallelThreshold(long) parallel threshold} and part size proportionately.

      • doSingleDownload

        protected void doSingleDownload()
                         throws java.io.IOException
        Throws:
        java.io.IOException

      • doParallelDownload

        protected void doParallelDownload()
                           throws java.lang.Exception
        Throws:
        java.lang.Exception

      • getS3Client

        public S3Client getS3Client()

      • getBucket

        public java.lang.String getBucket()

      • getKey

        public java.lang.String getKey()

      • getFile

        public java.io.File getFile()

      • getObjectSize

        public java.lang.Long getObjectSize()

      • getBytesTransferred

        public long getBytesTransferred()

      • getParallelThreshold

        public long getParallelThreshold()

      • setParallelThreshold

        public void setParallelThreshold​(long parallelThreshold)
        Sets the threshold above which parallel operations are used to download, and below which a single-GET is used. This only applies when using the download() method. Note the default threshold is DEFAULT_PARALLEL_THRESHOLD

      • getPartSize

        public long getPartSize()

      • setPartSize

        public void setPartSize​(long partSize)
        Sets the size of each part to download. Note that 1MB is the minimum part size and the default is DEFAULT_PART_SIZE.

      • getThreads

        public int getThreads()

      • setThreads

        public void setThreads​(int threads)
        Sets the number of threads to use for transferring parts. thread parts will be transferred in parallel. Default is 6

      • getExecutorService

        public java.util.concurrent.ExecutorService getExecutorService()

      • setExecutorService

        public void setExecutorService​(java.util.concurrent.ExecutorService executorService)
        Allows for providing a custom thread executor (i.e. for custom thread factories). Note that if you set a custom executor service, the threads property will be ignored.

      • setProgressListener

        public void setProgressListener​(ProgressListener progressListener)

      • withParallelThreshold

        public LargeFileDownloader withParallelThreshold​(long parallelThreshold)

      • withExecutorService

        public LargeFileDownloader withExecutorService​(java.util.concurrent.ExecutorService executorService)