Package com.emc.object.s3

Class LargeFileUploader

  • All Implemented Interfaces:
    ProgressListener, java.lang.Runnable
    public class LargeFileUploader
extends java.lang.Object
implements java.lang.Runnable, ProgressListener
    Convenience class to facilitate multipart upload for large files. This class will split the file and upload it in parts, transferring several parts simultaneously to maximize efficiency. If any errors occur during the upload, the target object will not exist (byte-range uploaded objects will be deleted and any created MPU will be aborted), unless the upload is an MPU and isAbortMpuOnFailure() is false, in which case, the MPU will be preserved.

    • Constructor Detail

      • LargeFileUploader

        public LargeFileUploader​(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         java.io.File file)
        Creates a new LargeFileUpload instance using the specified s3Client to upload file to bucket/key.

      • LargeFileUploader

        public LargeFileUploader​(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         java.io.InputStream stream,
                         long size)
        Creates a new LargeFileUpload instance using the specified s3Client to upload from a single stream to bucket/key. Note that this type of upload is single-threaded and not very efficient.

      • LargeFileUploader

        public LargeFileUploader​(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         LargeFileMultipartSource multipartSource)
        Creates a new LargeFileUpload instance using the specified s3Client to upload from a multipartSource to bucket/key.
        See Also:
        LargeFileMultipartSource

      • LargeFileUploader

        public LargeFileUploader​(S3Client s3Client,
                         java.lang.String srcBucket,
                         java.lang.String srcKey,
                         java.lang.String dstBucket,
                         java.lang.String dstKey)
        Creates a new LargeFileUpload instance using the specified s3Client to copy from srcBucket/srcKey to dstBucket/dstKey without streaming data between the client and ECS server.

    • Method Detail

      • getMpuETag

        public static java.lang.String getMpuETag​(java.util.List<MultipartPartETag> partETags)

      • progress

        public void progress​(long completed,
                     long total)
        Description copied from interface: ProgressListener
        Provides feedback on the number of bytes completed and the total number of bytes to transfer.
        Specified by:
        progress in interface ProgressListener
        Parameters:
        completed - bytes completely transferred
        total - total number of bytes to transfer

      • transferred

        public void transferred​(long size)
        Description copied from interface: ProgressListener
        Reports that some bytes have been transferred. This is a raw method that will be called frequently and can be used for computing current transfer rate. Note that if data is retried, the sum of this method's events may be more than the total object size. For reporting on percent complete, use the progress method instead.
        Specified by:
        transferred in interface ProgressListener
        Parameters:
        size - number of bytes transferred

      • run

        public void run()
        Specified by:
        run in interface java.lang.Runnable

      • getMinPartSize

        protected long getMinPartSize()

      • putObject

        protected java.lang.String putObject​(java.io.InputStream is)

      • listParts

        protected java.util.List<MultipartPart> listParts​(java.lang.String uploadId)

      • initMpu

        protected java.lang.String initMpu()

      • uploadPart

        protected MultipartPartETag uploadPart​(java.lang.String uploadId,
                                       int partNumber,
                                       java.io.InputStream is,
                                       long length)

      • abortMpu

        protected void abortMpu​(java.lang.String uploadId)

      • uploadAsync

        public LargeFileUpload uploadAsync()
        This async version of upload() will start the upload process in the background and immediately return a LargeFileUpload instance. This allows pausing or aborting the upload in the middle, or you can use waitForCompletion() to block until the upload is complete.
        See Also:
        upload()

      • upload

        public void upload()
        This method will automatically choose between MPU and single-PUT operations based on a configured threshold. Note the default threshold is DEFAULT_MPU_THRESHOLD. Also note that the defaults in this class are optimized for high-speed LAN connectivity. When operating over a WAN or a slower connection, you should reduce the MPU threshold and part size proportionately.

      • monitorStream

        protected java.io.InputStream monitorStream​(java.io.InputStream stream)

      • doSinglePut

        public void doSinglePut()

      • doSingleCopy

        public void doSingleCopy()

      • doMultipartUpload

        public void doMultipartUpload()

      • doByteRangeUpload

        public void doByteRangeUpload()

      • configure

        protected void configure()
        This method should be idempotent

      • getS3Client

        public S3Client getS3Client()

      • getBucket

        public java.lang.String getBucket()

      • getKey

        public java.lang.String getKey()

      • getStream

        public java.io.InputStream getStream()

      • getFullSize

        public long getFullSize()

      • getBytesTransferred

        public long getBytesTransferred()

      • getETag

        public java.lang.String getETag()
        Returns the result ETag after an upload is successfully complete.

      • getVersionId

        public java.lang.String getVersionId()
        Returns the result versionId after an upload is successfully completed to a version-enabled bucket.

      • setObjectMetadata

        public void setObjectMetadata​(S3ObjectMetadata objectMetadata)

      • getCannedAcl

        public CannedAcl getCannedAcl()

      • setCannedAcl

        public void setCannedAcl​(CannedAcl cannedAcl)

      • isCloseStream

        public boolean isCloseStream()

      • setCloseStream

        public void setCloseStream​(boolean closeStream)

      • getMpuThreshold

        public long getMpuThreshold()

      • setMpuThreshold

        public void setMpuThreshold​(long mpuThreshold)
        Sets the threshold above which an MPU operation is used to upload, and below which a single-PUT is used. This only applies when using the upload() method. Note the default threshold is DEFAULT_MPU_THRESHOLD

      • getPartSize

        public long getPartSize()

      • setPartSize

        public void setPartSize​(long partSize)
        Sets the size of each part to upload. Note the default part size is DEFAULT_PART_SIZE and MIN_PART_SIZE is the minimum part size. Note also there is a maximum of 10,000 parts, and the part size will be increased automatically if necessary.

      • getThreads

        public int getThreads()

      • setThreads

        public void setThreads​(int threads)
        Sets the number of threads to use for transferring parts. thread parts will be transferred in parallel. Default is 6

      • getExecutorService

        public java.util.concurrent.ExecutorService getExecutorService()

      • setExecutorService

        public void setExecutorService​(java.util.concurrent.ExecutorService executorService)
        Allows for providing a custom thread executor (i.e. for custom thread factories). Note that if you set a custom executor service, the threads property will be ignored.

      • setProgressListener

        public void setProgressListener​(ProgressListener progressListener)

      • getResumeContext

        public LargeFileUploaderResumeContext getResumeContext()
        During an upload operation, the resumeContext is kept up-to-date with the uploadId and list of uploaded parts.

      • setResumeContext

        public void setResumeContext​(LargeFileUploaderResumeContext resumeContext)
        Use when resuming an existing incomplete MPU by skipping existing parts. To resume an MPU, you *must* provide an uploadId to resume. If a part list is not provided here, the uploadId parts will be listed to find existing parts to skip. All parts in the existing upload must conform to the expected part size and count, based on setMpuThreshold(long) and setPartSize(long).
        Throws:
        S3Exception - if the provided uploadId does not exist, or any other S3 errors occur
        java.lang.IllegalArgumentException - if the uploadId is null or any of the parts are invalid
        java.lang.UnsupportedOperationException - if the size of the source is *not* above mpuThreshold
        See Also:
        LargeFileUploaderResumeContext

      • isAbortMpuOnFailure

        public boolean isAbortMpuOnFailure()

      • setAbortMpuOnFailure

        public void setAbortMpuOnFailure​(boolean abortMpuOnFailure)
        Specifies whether MPU is aborted with any failure If a failure occurs and abortMpuOnFailure is true, then MPU is aborted and the resumeContext is cleared (uploadId and uploadedParts are set to null). If abortMpuOnFailure is false, MPU is left intact and the resumeContext could have a list successfully uploaded parts.

      • getSourceVersionId

        public java.lang.String getSourceVersionId()

      • setSourceVersionId

        public void setSourceVersionId​(java.lang.String sourceVersionId)

      • withCloseStream

        public LargeFileUploader withCloseStream​(boolean closeStream)

      • withMpuThreshold

        public LargeFileUploader withMpuThreshold​(long mpuThreshold)

      • withExecutorService

        public LargeFileUploader withExecutorService​(java.util.concurrent.ExecutorService executorService)

      • withSourceVersionId

        public LargeFileUploader withSourceVersionId​(java.lang.String sourceVersionId)