LargeFileUploader (object-client 3.4.11 API)

java.lang.Object
- com.emc.object.s3.LargeFileUploader

All Implemented Interfaces:

ProgressListener, java.lang.Runnable
```
public class LargeFileUploader
extends java.lang.Object
implements java.lang.Runnable, ProgressListener
```
Convenience class to facilitate multipart upload for large files. This class will split the file and upload it in parts, transferring several parts simultaneously to maximize efficiency. If any errors occur during the upload, the target object will not exist (byte-range uploaded objects will be deleted and any created MPU will be aborted), unless the upload is an MPU and isAbortMpuOnFailure() is false, in which case, the MPU will be preserved.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

protected class LargeFileUploader.PutObjectTask

protected class LargeFileUploader.VerifySourcePartTask

Nested Classes
Modifier and Type	Class and Description
`protected class`	`LargeFileUploader.PutObjectTask`
`protected class`	`LargeFileUploader.VerifySourcePartTask`

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`DEFAULT_MPU_THRESHOLD`
`static long`	`DEFAULT_PART_SIZE`
`static int`	`DEFAULT_THREADS`
`static int`	`MAX_PARTS`
`static long`	`MIN_PART_SIZE`

Constructor Summary

Constructors
Constructor and Description
`LargeFileUploader(S3Client s3Client, java.lang.String bucket, java.lang.String key, java.io.File file)` Creates a new LargeFileUpload instance using the specified `s3Client` to upload `file` to `bucket/key`.
`LargeFileUploader(S3Client s3Client, java.lang.String bucket, java.lang.String key, java.io.InputStream stream, long size)` Creates a new LargeFileUpload instance using the specified `s3Client` to upload from a single `stream` to `bucket/key`.
`LargeFileUploader(S3Client s3Client, java.lang.String bucket, java.lang.String key, LargeFileMultipartSource multipartSource)` Creates a new LargeFileUpload instance using the specified `s3Client` to upload from a `multipartSource` to `bucket/key`.
`LargeFileUploader(S3Client s3Client, java.lang.String srcBucket, java.lang.String srcKey, java.lang.String dstBucket, java.lang.String dstKey)` Creates a new LargeFileUpload instance using the specified `s3Client` to copy from `srcBucket/srcKey` to `dstBucket/dstKey` without streaming data between the client and ECS server.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`abortMpu(java.lang.String uploadId)`
`protected CompleteMultipartUploadResult`	`completeMpu(java.lang.String uploadId, java.util.SortedSet<MultipartPartETag> parts)`
`protected void`	`configure()` This method should be idempotent
`void`	`doByteRangeUpload()`
`void`	`doMultipartUpload()`
`void`	`doSingleCopy()`
`void`	`doSinglePut()`
`AccessControlList`	`getAcl()`
`java.lang.String`	`getBucket()`
`long`	`getBytesTransferred()`
`CannedAcl`	`getCannedAcl()`
`java.lang.String`	`getETag()` Returns the result ETag after an upload is successfully complete.
`java.util.concurrent.ExecutorService`	`getExecutorService()`
`long`	`getFullSize()`
`java.lang.String`	`getKey()`
`protected long`	`getMinPartSize()`
`static java.lang.String`	`getMpuETag(java.util.List<MultipartPartETag> partETags)`
`long`	`getMpuThreshold()`
`LargeFileMultipartSource`	`getMultipartSource()`
`S3ObjectMetadata`	`getObjectMetadata()`
`long`	`getPartSize()`
`ProgressListener`	`getProgressListener()`
`LargeFileUploaderResumeContext`	`getResumeContext()` During an upload operation, the `resumeContext` is kept up-to-date with the uploadId and list of uploaded parts.
`S3Client`	`getS3Client()`
`java.lang.String`	`getSourceVersionId()`
`java.io.InputStream`	`getStream()`
`int`	`getThreads()`
`java.lang.String`	`getVersionId()` Returns the result versionId after an upload is successfully completed to a version-enabled bucket.
`protected java.lang.String`	`initMpu()`
`boolean`	`isAbortMpuOnFailure()`
`boolean`	`isCloseStream()`
`protected java.util.List<MultipartPart>`	`listParts(java.lang.String uploadId)`
`protected java.io.InputStream`	`monitorStream(java.io.InputStream stream)`
`void`	`progress(long completed, long total)` Provides feedback on the number of bytes completed and the total number of bytes to transfer.
`protected java.lang.String`	`putObject(java.io.InputStream is)`
`void`	`run()`
`void`	`setAbortMpuOnFailure(boolean abortMpuOnFailure)` Specifies whether MPU is aborted with any failure If a failure occurs and abortMpuOnFailure is true, then MPU is aborted and the resumeContext is cleared (uploadId and uploadedParts are set to null).
`void`	`setAcl(AccessControlList acl)`
`void`	`setCannedAcl(CannedAcl cannedAcl)`
`void`	`setCloseStream(boolean closeStream)`
`void`	`setExecutorService(java.util.concurrent.ExecutorService executorService)` Allows for providing a custom thread executor (i.e.
`void`	`setMpuThreshold(long mpuThreshold)` Sets the threshold above which an MPU operation is used to upload, and below which a single-PUT is used.
`void`	`setObjectMetadata(S3ObjectMetadata objectMetadata)`
`void`	`setPartSize(long partSize)` Sets the size of each part to upload.
`void`	`setProgressListener(ProgressListener progressListener)`
`void`	`setResumeContext(LargeFileUploaderResumeContext resumeContext)` Use when resuming an existing incomplete MPU by skipping existing parts.
`void`	`setSourceVersionId(java.lang.String sourceVersionId)`
`void`	`setThreads(int threads)` Sets the number of threads to use for transferring parts.
`void`	`transferred(long size)` Reports that some bytes have been transferred.
`void`	`upload()` This method will automatically choose between MPU and single-PUT operations based on a configured threshold.
`LargeFileUpload`	`uploadAsync()` This async version of upload() will start the upload process in the background and immediately return a `LargeFileUpload` instance.
`protected MultipartPartETag`	`uploadPart(java.lang.String uploadId, int partNumber, java.io.InputStream is, long length)`
`LargeFileUploader`	`withAbortMpuOnFailure(boolean abortMpuOnFailure)`
`LargeFileUploader`	`withAcl(AccessControlList acl)`
`LargeFileUploader`	`withCannedAcl(CannedAcl cannedAcl)`
`LargeFileUploader`	`withCloseStream(boolean closeStream)`
`LargeFileUploader`	`withExecutorService(java.util.concurrent.ExecutorService executorService)`
`LargeFileUploader`	`withMpuThreshold(long mpuThreshold)`
`LargeFileUploader`	`withObjectMetadata(S3ObjectMetadata objectMetadata)`
`LargeFileUploader`	`withPartSize(java.lang.Long partSize)`
`LargeFileUploader`	`withProgressListener(ProgressListener progressListener)`
`LargeFileUploader`	`withResumeContext(LargeFileUploaderResumeContext resumeContext)`
`LargeFileUploader`	`withSourceVersionId(java.lang.String sourceVersionId)`
`LargeFileUploader`	`withThreads(int threads)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEFAULT_THREADS
```
public static final int DEFAULT_THREADS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_MPU_THRESHOLD
```
public static final int DEFAULT_MPU_THRESHOLD
```
    See Also:
    
    Constant Field Values
  - MIN_PART_SIZE
```
public static final long MIN_PART_SIZE
```
    See Also:
    
    Constant Field Values
  - DEFAULT_PART_SIZE
```
public static final long DEFAULT_PART_SIZE
```
    See Also:
    
    Constant Field Values
  - MAX_PARTS
```
public static final int MAX_PARTS
```
    See Also:
    
    Constant Field Values
- Constructor Detail
  - LargeFileUploader
```
public LargeFileUploader(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         java.io.File file)
```
    Creates a new LargeFileUpload instance using the specified s3Client to upload file to bucket/key.
  - LargeFileUploader
```
public LargeFileUploader(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         java.io.InputStream stream,
                         long size)
```
    Creates a new LargeFileUpload instance using the specified s3Client to upload from a single stream to bucket/key. Note that this type of upload is single-threaded and not very efficient.
  - LargeFileUploader
```
public LargeFileUploader(S3Client s3Client,
                         java.lang.String bucket,
                         java.lang.String key,
                         LargeFileMultipartSource multipartSource)
```
    Creates a new LargeFileUpload instance using the specified s3Client to upload from a multipartSource to bucket/key.
    
    See Also:
    
    LargeFileMultipartSource
  - LargeFileUploader
```
public LargeFileUploader(S3Client s3Client,
                         java.lang.String srcBucket,
                         java.lang.String srcKey,
                         java.lang.String dstBucket,
                         java.lang.String dstKey)
```
    Creates a new LargeFileUpload instance using the specified s3Client to copy from srcBucket/srcKey to dstBucket/dstKey without streaming data between the client and ECS server.
- Method Detail
  - getMpuETag
```
public static java.lang.String getMpuETag(java.util.List<MultipartPartETag> partETags)
```
  - progress
```
public void progress(long completed,
                     long total)
```
    Description copied from interface: ProgressListener
    
    Provides feedback on the number of bytes completed and the total number of bytes to transfer.
    
    Specified by:
    
    progress in interface ProgressListener
    
    Parameters:
    
    completed - bytes completely transferred
    
    total - total number of bytes to transfer
  - transferred
```
public void transferred(long size)
```
    Description copied from interface: ProgressListener
    
    Reports that some bytes have been transferred. This is a raw method that will be called frequently and can be used for computing current transfer rate. Note that if data is retried, the sum of this method's events may be more than the total object size. For reporting on percent complete, use the progress method instead.
    
    Specified by:
    
    transferred in interface ProgressListener
    
    Parameters:
    
    size - number of bytes transferred
  - run
```
public void run()
```
    Specified by:
    
    run in interface java.lang.Runnable
  - getMinPartSize
```
protected long getMinPartSize()
```
  - putObject
```
protected java.lang.String putObject(java.io.InputStream is)
```
  - listParts
```
protected java.util.List<MultipartPart> listParts(java.lang.String uploadId)
```
  - initMpu
```
protected java.lang.String initMpu()
```
  - uploadPart
```
protected MultipartPartETag uploadPart(java.lang.String uploadId,
                                       int partNumber,
                                       java.io.InputStream is,
                                       long length)
```
  - completeMpu
```
protected CompleteMultipartUploadResult completeMpu(java.lang.String uploadId,
                                                    java.util.SortedSet<MultipartPartETag> parts)
```
  - abortMpu
```
protected void abortMpu(java.lang.String uploadId)
```
  - uploadAsync
```
public LargeFileUpload uploadAsync()
```
    This async version of upload() will start the upload process in the background and immediately return a LargeFileUpload instance. This allows pausing or aborting the upload in the middle, or you can use waitForCompletion() to block until the upload is complete.
    
    See Also:
    
    upload()
  - upload
```
public void upload()
```
    This method will automatically choose between MPU and single-PUT operations based on a configured threshold. Note the default threshold is DEFAULT_MPU_THRESHOLD. Also note that the defaults in this class are optimized for high-speed LAN connectivity. When operating over a WAN or a slower connection, you should reduce the MPU threshold and part size proportionately.
  - monitorStream
```
protected java.io.InputStream monitorStream(java.io.InputStream stream)
```
  - doSinglePut
```
public void doSinglePut()
```
  - doSingleCopy
```
public void doSingleCopy()
```
  - doMultipartUpload
```
public void doMultipartUpload()
```
  - doByteRangeUpload
```
public void doByteRangeUpload()
```
  - configure
```
protected void configure()
```
    This method should be idempotent
  - getS3Client
```
public S3Client getS3Client()
```
  - getBucket
```
public java.lang.String getBucket()
```
  - getKey
```
public java.lang.String getKey()
```
  - getStream
```
public java.io.InputStream getStream()
```
  - getMultipartSource
```
public LargeFileMultipartSource getMultipartSource()
```
  - getFullSize
```
public long getFullSize()
```
  - getBytesTransferred
```
public long getBytesTransferred()
```
  - getETag
```
public java.lang.String getETag()
```
    Returns the result ETag after an upload is successfully complete.
  - getVersionId
```
public java.lang.String getVersionId()
```
    Returns the result versionId after an upload is successfully completed to a version-enabled bucket.
  - getObjectMetadata
```
public S3ObjectMetadata getObjectMetadata()
```
  - setObjectMetadata
```
public void setObjectMetadata(S3ObjectMetadata objectMetadata)
```
  - getAcl
```
public AccessControlList getAcl()
```
  - setAcl
```
public void setAcl(AccessControlList acl)
```
  - getCannedAcl
```
public CannedAcl getCannedAcl()
```
  - setCannedAcl
```
public void setCannedAcl(CannedAcl cannedAcl)
```
  - isCloseStream
```
public boolean isCloseStream()
```
  - setCloseStream
```
public void setCloseStream(boolean closeStream)
```
  - getMpuThreshold
```
public long getMpuThreshold()
```
  - setMpuThreshold
```
public void setMpuThreshold(long mpuThreshold)
```
    Sets the threshold above which an MPU operation is used to upload, and below which a single-PUT is used. This only applies when using the upload() method. Note the default threshold is DEFAULT_MPU_THRESHOLD
  - getPartSize
```
public long getPartSize()
```
  - setPartSize
```
public void setPartSize(long partSize)
```
    Sets the size of each part to upload. Note the default part size is DEFAULT_PART_SIZE and MIN_PART_SIZE is the minimum part size. Note also there is a maximum of 10,000 parts, and the part size will be increased automatically if necessary.
  - getThreads
```
public int getThreads()
```
  - setThreads
```
public void setThreads(int threads)
```
    Sets the number of threads to use for transferring parts. thread parts will be transferred in parallel. Default is 6
  - getExecutorService
```
public java.util.concurrent.ExecutorService getExecutorService()
```
  - setExecutorService
```
public void setExecutorService(java.util.concurrent.ExecutorService executorService)
```
    Allows for providing a custom thread executor (i.e. for custom thread factories). Note that if you set a custom executor service, the threads property will be ignored.
  - getProgressListener
```
public ProgressListener getProgressListener()
```
  - setProgressListener
```
public void setProgressListener(ProgressListener progressListener)
```
  - getResumeContext
```
public LargeFileUploaderResumeContext getResumeContext()
```
    During an upload operation, the resumeContext is kept up-to-date with the uploadId and list of uploaded parts.
  - setResumeContext
```
public void setResumeContext(LargeFileUploaderResumeContext resumeContext)
```
    Use when resuming an existing incomplete MPU by skipping existing parts. To resume an MPU, you *must* provide an uploadId to resume. If a part list is not provided here, the uploadId parts will be listed to find existing parts to skip. All parts in the existing upload must conform to the expected part size and count, based on setMpuThreshold(long) and setPartSize(long).
    
    Throws:
    
    S3Exception - if the provided uploadId does not exist, or any other S3 errors occur
    
    java.lang.IllegalArgumentException - if the uploadId is null or any of the parts are invalid
    
    java.lang.UnsupportedOperationException - if the size of the source is *not* above mpuThreshold
    
    See Also:
    
    LargeFileUploaderResumeContext
  - isAbortMpuOnFailure
```
public boolean isAbortMpuOnFailure()
```
  - setAbortMpuOnFailure
```
public void setAbortMpuOnFailure(boolean abortMpuOnFailure)
```
    Specifies whether MPU is aborted with any failure If a failure occurs and abortMpuOnFailure is true, then MPU is aborted and the resumeContext is cleared (uploadId and uploadedParts are set to null). If abortMpuOnFailure is false, MPU is left intact and the resumeContext could have a list successfully uploaded parts.
  - getSourceVersionId
```
public java.lang.String getSourceVersionId()
```
  - setSourceVersionId
```
public void setSourceVersionId(java.lang.String sourceVersionId)
```
  - withObjectMetadata
```
public LargeFileUploader withObjectMetadata(S3ObjectMetadata objectMetadata)
```
  - withAcl
```
public LargeFileUploader withAcl(AccessControlList acl)
```
  - withCannedAcl
```
public LargeFileUploader withCannedAcl(CannedAcl cannedAcl)
```
  - withCloseStream
```
public LargeFileUploader withCloseStream(boolean closeStream)
```
  - withMpuThreshold
```
public LargeFileUploader withMpuThreshold(long mpuThreshold)
```
  - withPartSize
```
public LargeFileUploader withPartSize(java.lang.Long partSize)
```
  - withThreads
```
public LargeFileUploader withThreads(int threads)
```
  - withExecutorService
```
public LargeFileUploader withExecutorService(java.util.concurrent.ExecutorService executorService)
```
  - withProgressListener
```
public LargeFileUploader withProgressListener(ProgressListener progressListener)
```
  - withResumeContext
```
public LargeFileUploader withResumeContext(LargeFileUploaderResumeContext resumeContext)
```
    See Also:
    
    setResumeContext(LargeFileUploaderResumeContext)
  - withAbortMpuOnFailure
```
public LargeFileUploader withAbortMpuOnFailure(boolean abortMpuOnFailure)
```
    See Also:
    
    setAbortMpuOnFailure(boolean)
  - withSourceVersionId
```
public LargeFileUploader withSourceVersionId(java.lang.String sourceVersionId)
```

Class LargeFileUploader

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_THREADS

DEFAULT_MPU_THRESHOLD

MIN_PART_SIZE

DEFAULT_PART_SIZE

MAX_PARTS

Constructor Detail

LargeFileUploader

LargeFileUploader

LargeFileUploader

LargeFileUploader

Method Detail

getMpuETag

progress

transferred

run

getMinPartSize

putObject

listParts

initMpu

uploadPart

completeMpu

abortMpu

uploadAsync

upload

monitorStream

doSinglePut

doSingleCopy

doMultipartUpload

doByteRangeUpload

configure

getS3Client

getBucket

getKey

getStream

getMultipartSource

getFullSize

getBytesTransferred

getETag

getVersionId

getObjectMetadata

setObjectMetadata

getAcl

setAcl

getCannedAcl

setCannedAcl

isCloseStream

setCloseStream

getMpuThreshold

setMpuThreshold

getPartSize

setPartSize

getThreads

setThreads

getExecutorService

setExecutorService

getProgressListener

setProgressListener

getResumeContext

setResumeContext

isAbortMpuOnFailure

setAbortMpuOnFailure

getSourceVersionId

setSourceVersionId

withObjectMetadata

withAcl

withCannedAcl

withCloseStream

withMpuThreshold

withPartSize

withThreads

withExecutorService

withProgressListener

withResumeContext