WriterImpl (ORC Core 1.6.5 API)
Date: 2020-10-01

org.apache.orc.impl

## Class WriterImpl

- java.lang.Object
- - org.apache.orc.impl.WriterImpl

- All Implemented Interfaces:
    Closeable, AutoCloseable, WriterInternal, MemoryManager.Callback, Writer
  Direct Known Subclasses:
    WriterImplV2

      public class WriterImpl
      extends Object
      implements WriterInternal, MemoryManager.Callback

  An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams. This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread. There are no known cases where these happen between different threads today. Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `WriterImpl(FileSystem fs, Path path, OrcFile.WriterOptions opts)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addRowBatch(VectorizedRowBatch batch)` Add a row batch to the ORC file.
    `void` | `addUserMetadata(String name, ByteBuffer value)` Add arbitrary meta-data to the ORC file.
    `void` | `appendStripe(byte[] stripe, int offset, int length, StripeInformation stripeInfo, OrcProto.StripeStatistics stripeStatistics)` Fast stripe append to ORC file.
    `void` | `appendStripe(byte[] stripe, int offset, int length, StripeInformation stripeInfo, StripeStatistics[] stripeStatistics)` Fast stripe append to ORC file.
    `void` | `appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)` Update the current user metadata with a list of new values.
    `boolean` | `checkMemory(double newScale)` The scale factor for the stripe size has changed and thus the writer should adjust their desired size appropriately.
    `void` | `close()` Flush all of the buffers and close the file.
    `static CompressionCodec` | `createCodec(CompressionKind kind)`
    `CompressionCodec` | `getCompressionCodec()`
    `static int` | `getEstimatedBufferSize(long stripeSize, int numColumns, int bs)`
    `long` | `getNumberOfRows()` Row count gets updated when flushing the stripes.
    `long` | `getRawDataSize()` Raw data size will be compute when writing the file footer.
    `TypeDescription` | `getSchema()` Get the schema for this writer
    `ColumnStatistics[]` | `getStatistics()` Get the statistics about the columns in the file.
    `void` | `increaseCompressionSize(int newSize)` Increase the buffer size for this writer.
    `long` | `writeIntermediateFooter()` Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### WriterImpl

          public WriterImpl(FileSystem fs,
                            Path path,
                            OrcFile.WriterOptions opts)
                     throws IOException

      Throws:
        `IOException`

  - ### Method Detail

    - #### getEstimatedBufferSize

          public static int getEstimatedBufferSize(long stripeSize,
                                                   int numColumns,
                                                   int bs)

    - #### increaseCompressionSize

          public void increaseCompressionSize(int newSize)

      Description copied from interface: `WriterInternal`
      Increase the buffer size for this writer. This function is internal only and should only be called by the ORC file merger.
      Specified by:
        `increaseCompressionSize` in interface `WriterInternal`
      Parameters:
        `newSize` - the new buffer size.

    - #### createCodec

          public static CompressionCodec createCodec(CompressionKind kind)

    - #### checkMemory

          public boolean checkMemory(double newScale)
                              throws IOException

      Description copied from interface: `MemoryManager.Callback`
      The scale factor for the stripe size has changed and thus the writer should adjust their desired size appropriately.
      Specified by:
        `checkMemory` in interface `MemoryManager.Callback`
      Parameters:
        `newScale` - the current scale factor for memory allocations
      Returns:
        true if the writer was over the limit
      Throws:
        `IOException`

    - #### getSchema

          public TypeDescription getSchema()

      Description copied from interface: `Writer`
      Get the schema for this writer
      Specified by:
        `getSchema` in interface `Writer`
      Returns:
        the file schema

    - #### addUserMetadata

          public void addUserMetadata(String name,
                                      ByteBuffer value)

      Description copied from interface: `Writer`
      Add arbitrary meta-data to the ORC file. This may be called at any point until the Writer is closed. If the same key is passed a second time, the second value will replace the first.
      Specified by:
        `addUserMetadata` in interface `Writer`
      Parameters:
        `name` - a key to label the data with.
        `value` - the contents of the metadata.

    - #### addRowBatch

          public void addRowBatch(VectorizedRowBatch batch)
                           throws IOException

      Description copied from interface: `Writer`
      Add a row batch to the ORC file.
      Specified by:
        `addRowBatch` in interface `Writer`
      Parameters:
        `batch` - the rows to add
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Description copied from interface: `Writer`
      Flush all of the buffers and close the file. No methods on this writer should be called afterwards.
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in interface `Writer`
      Throws:
        `IOException`

    - #### getRawDataSize

          public long getRawDataSize()

      Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.
      Specified by:
        `getRawDataSize` in interface `Writer`
      Returns:
        raw data size

    - #### getNumberOfRows

          public long getNumberOfRows()

      Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.
      Specified by:
        `getNumberOfRows` in interface `Writer`
      Returns:
        row count

    - #### writeIntermediateFooter

          public long writeIntermediateFooter()
                                       throws IOException

      Description copied from interface: `Writer`
      Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.
      Specified by:
        `writeIntermediateFooter` in interface `Writer`
      Returns:
        the offset that would be a valid end location for an ORC file
      Throws:
        `IOException`

    - #### appendStripe

          public void appendStripe(byte[] stripe,
                                   int offset,
                                   int length,
                                   StripeInformation stripeInfo,
                                   OrcProto.StripeStatistics stripeStatistics)
                            throws IOException

      Description copied from interface: `Writer`
      Fast stripe append to ORC file. This interface is used for fast ORC file merge with other ORC files. When merging, the file to be merged should pass stripe in binary form along with stripe information and stripe statistics. After appending last stripe of a file, use appendUserMetadata() to append any user metadata. This form only supports files with no column encryption. Use `Writer.appendStripe(byte[], int, int, StripeInformation, StripeStatistics[])` for files with encryption.
      Specified by:
        `appendStripe` in interface `Writer`
      Parameters:
        `stripe` - - stripe as byte array
        `offset` - - offset within byte array
        `length` - - length of stripe within byte array
        `stripeInfo` - - stripe information
        `stripeStatistics` - - unencrypted stripe statistics
      Throws:
        `IOException`

    - #### appendStripe

          public void appendStripe(byte[] stripe,
                                   int offset,
                                   int length,
                                   StripeInformation stripeInfo,
                                   StripeStatistics[] stripeStatistics)
                            throws IOException

      Description copied from interface: `Writer`
      Fast stripe append to ORC file. This interface is used for fast ORC file merge with other ORC files. When merging, the file to be merged should pass stripe in binary form along with stripe information and stripe statistics. After appending last stripe of a file, use `Writer.addUserMetadata(String, ByteBuffer)` to append any user metadata.
      Specified by:
        `appendStripe` in interface `Writer`
      Parameters:
        `stripe` - - stripe as byte array
        `offset` - - offset within byte array
        `length` - - length of stripe within byte array
        `stripeInfo` - - stripe information
        `stripeStatistics` - - stripe statistics with the last one being for the unencrypted data and the others being for each encryption variant.
      Throws:
        `IOException`

    - #### appendUserMetadata

          public void appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)

      Description copied from interface: `Writer`
      Update the current user metadata with a list of new values.
      Specified by:
        `appendUserMetadata` in interface `Writer`
      Parameters:
        `userMetadata` - - user metadata

    - #### getStatistics

          public ColumnStatistics[] getStatistics()

      Description copied from interface: `Writer`
      Get the statistics about the columns in the file. The output of this is based on the time at which it is called. It shall use all of the currently written data to provide the statistics. Please note there are costs involved with invoking this method and should be used judiciously.
      Specified by:
        `getStatistics` in interface `Writer`
      Returns:
        the information about the column

    - #### getCompressionCodec

          public CompressionCodec getCompressionCodec()

Copyright © 2013–2020 The Apache Software Foundation. All rights reserved.