Class AsynchronousLogShipper

  • All Implemented Interfaces:
    java.lang.Runnable, LogShipper

    public class AsynchronousLogShipper
    extends java.lang.Thread
    implements LogShipper

    Does asynchronous shipping of log records from the master to the slave being replicated to. The implementation does not ship log records as soon as they become available in the log buffer (synchronously), instead it does log shipping in the following two-fold scenarios 1) Periodically (i.e.) at regular intervals of time. 2) when a request is sent from the master controller (force flushing of the log buffer). 3) when a notification is received from the log shipper about a log buffer element becoming full and the load on the log buffer so warrants a ship.

    • Nested Class Summary

      • Nested classes/interfaces inherited from class java.lang.Thread

        java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int DEFAULT_FORCEFLUSH_TIMEOUT
      The number of millis a call to forceFlush will wait before giving up sending a chunk of log to the slave
      private ReplicationMessage failedChunk
      Store the log chunk that failed during a previous shipping attempt so that it can be re-shipped to the slave.
      private long failedChunkHighestInstant
      The highest log instant in failedChunk
      private static int FI_HIGH
      Fill information value indicative of a high load in the log buffer.
      private static int FI_LOW
      Fill information value indicative of a low load in the log buffer.
      private java.lang.Object forceFlushSemaphore
      Used to synchronize forceFlush calls
      private long highestShippedInstant
      The highest log instant shipped so far
      private long lastShippingTime
      Will store the time at which the last shipping happened.
      private ReplicationLogBuffer logBuffer
      Replication log buffer that contains the log records that need to be transmitted to the slave.
      private MasterController masterController
      The master controller that initialized this log shipper.
      private static long MAX
      If the fill information is less than FI_LOW the log shipper will ship with a MAX ms delay or when a buffer becomes full whichever comes first.
      private long maxShippingInterval
      Minimum interval (in milliseconds) between log shipping.
      private static long MIN
      If the fill information (obtained from the log buffer) is less than FI_HIGH but greater than FI_LOW the log shipper will ship with a MIN ms delay.
      private long minShippingInterval
      Minimum interval (in milliseconds) between log shipping.
      private java.lang.Object objLSTSync
      Object used to synchronize on while the log shipper thread is moved into the wait state, or while notifying it.
      private ReplicationLogger repLogger  
      private long shippingInterval
      Time interval (in milliseconds) at which the log shipping takes place.
      private boolean stopShipping
      Indicates whether a stop shipping request has been sent.
      private ReplicationMessageTransmit transmitter
      Replication message transmitter that is used for the network transmission of the log records retrieved from the log buffer (on the master) to the slave being replicated to.
      • Fields inherited from class java.lang.Thread

        MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      private long calculateSIfromFI()
      Will be used to calculate the shipping interval based on the fill information obtained from the log buffer.
      void flushBuffer()
      Transmits all the log records in the log buffer to the slave.
      void flushedInstance​(long latestInstanceFlushedToDisk)
      updates the information about the latest instance of the log record that has been flushed to the disk.
      void forceFlush()
      Transmits a chunk of log record from the log buffer to the slave, used by the master controller when the log buffer is full and some space needs to be freed for further log records.
      long getHighestShippedInstant()
      Get the highest log instant shipped so far
      private void getLogShipperProperties()
      Load relevant system properties: max and min log shipping interval
      void run()
      Ships log records from the log buffer to the slave being replicated to.
      private boolean shipALogChunk()
      Retrieves a chunk of log records, if available, from the log buffer and transmits them to the slave.
      void stopLogShipment()
      Stop shipping log records.
      void workToDo()
      Used to notify the log shipper that a log buffer element is full.
      • Methods inherited from class java.lang.Thread

        activeCount, checkAccess, clone, countStackFrames, currentThread, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, onSpinWait, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, suspend, toString, yield
      • Methods inherited from class java.lang.Object

        equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
    • Field Detail

      • logBuffer

        private final ReplicationLogBuffer logBuffer
        Replication log buffer that contains the log records that need to be transmitted to the slave.
      • transmitter

        private ReplicationMessageTransmit transmitter
        Replication message transmitter that is used for the network transmission of the log records retrieved from the log buffer (on the master) to the slave being replicated to.
      • shippingInterval

        private long shippingInterval
        Time interval (in milliseconds) at which the log shipping takes place.
      • minShippingInterval

        private long minShippingInterval
        Minimum interval (in milliseconds) between log shipping. Defaults to MIN, but can be configured using system property derby.replication.minLogShippingInterval
        See Also:
        MIN
      • maxShippingInterval

        private long maxShippingInterval
        Minimum interval (in milliseconds) between log shipping. Defaults to MAX, but can be configured using system property derby.replication.maxLogShippingInterval
        See Also:
        MAX
      • lastShippingTime

        private long lastShippingTime
        Will store the time at which the last shipping happened. Will be used to calculate the interval between the log ships upon receiving a notification from the log buffer.
      • stopShipping

        private volatile boolean stopShipping
        Indicates whether a stop shipping request has been sent. true - stop shipping log records false - shipping can continue without interruption.
      • masterController

        private MasterController masterController
        The master controller that initialized this log shipper.
      • objLSTSync

        private java.lang.Object objLSTSync
        Object used to synchronize on while the log shipper thread is moved into the wait state, or while notifying it.
      • forceFlushSemaphore

        private java.lang.Object forceFlushSemaphore
        Used to synchronize forceFlush calls
      • DEFAULT_FORCEFLUSH_TIMEOUT

        public static final int DEFAULT_FORCEFLUSH_TIMEOUT
        The number of millis a call to forceFlush will wait before giving up sending a chunk of log to the slave
        See Also:
        Constant Field Values
      • failedChunk

        private ReplicationMessage failedChunk
        Store the log chunk that failed during a previous shipping attempt so that it can be re-shipped to the slave.
      • failedChunkHighestInstant

        private long failedChunkHighestInstant
        The highest log instant in failedChunk
      • highestShippedInstant

        private long highestShippedInstant
        The highest log instant shipped so far
      • FI_LOW

        private static final int FI_LOW
        Fill information value indicative of a low load in the log buffer.
        See Also:
        Constant Field Values
      • FI_HIGH

        private static final int FI_HIGH
        Fill information value indicative of a high load in the log buffer.
        See Also:
        Constant Field Values
      • MIN

        private static final long MIN
        If the fill information (obtained from the log buffer) is less than FI_HIGH but greater than FI_LOW the log shipper will ship with a MIN ms delay. MIN is a value that is only as large as not to affect the performance of the master database significantly.
        See Also:
        Constant Field Values
      • MAX

        private static final long MAX
        If the fill information is less than FI_LOW the log shipper will ship with a MAX ms delay or when a buffer becomes full whichever comes first. The delay however will not be smaller than MIN. max(MAX, DEFAULT_NUMBER_LOG_BUFFERS*MIN) is the maximum delay between a log record is committed at the master until it is replicated to the slave. Hence the default latency should be atleast greater than the maximum latency offered by the choice of MIN, hence MAX > DEFAULT_NUMBER_LOG_BUFFERS*MIN.
        See Also:
        Constant Field Values
    • Constructor Detail

      • AsynchronousLogShipper

        public AsynchronousLogShipper​(ReplicationLogBuffer logBuffer,
                                      ReplicationMessageTransmit transmitter,
                                      MasterController masterController,
                                      ReplicationLogger repLogger)
        Constructor initializes the log buffer, the replication message transmitter, the shipping interval and the master controller.
        Parameters:
        logBuffer - the replication log buffer that contains the log record chunks to be transmitted to the slave.
        transmitter - the replication message transmitter that is used for network transmission of retrieved log records.
        masterController - The master controller that initialized this log shipper.
        repLogger - The replication logger that will write messages to the log file (typically derby.log)
    • Method Detail

      • run

        public void run()
        Ships log records from the log buffer to the slave being replicated to. The log shipping happens between shipping intervals of time, the shipping interval being derived from the fill information (an indicator of load in the log buffer) obtained from the log buffer. The shipping can also be triggered in the following situations, 1) Based on notifications from the log buffer, where the fill information is again used as the basis to decide whether a shipping should happen or not 2) On a forceFlush triggered by the log buffer becoming full and the LogBufferFullException being thrown.
        Specified by:
        run in interface java.lang.Runnable
        Overrides:
        run in class java.lang.Thread
      • shipALogChunk

        private boolean shipALogChunk()
                               throws java.io.IOException,
                                      StandardException
        Retrieves a chunk of log records, if available, from the log buffer and transmits them to the slave. Used for both periodic and forced shipping.
        Returns:
        true if a chunk of log records was shipped. false if no log records were shipped because log buffer is empty.
        Throws:
        java.io.IOException - If an exception occurs while trying to ship the replication message (containing the log records) across the network.
        StandardException - If an exception occurs while trying to read log records from the log buffer.
      • flushBuffer

        public void flushBuffer()
                         throws java.io.IOException,
                                StandardException
        Transmits all the log records in the log buffer to the slave.
        Specified by:
        flushBuffer in interface LogShipper
        Throws:
        java.io.IOException - If an exception occurs while trying to ship the replication message (containing the log records) across the network.
        StandardException - If an exception occurs while trying to read log records from the log buffer.
      • forceFlush

        public void forceFlush()
                        throws java.io.IOException,
                               StandardException
        Transmits a chunk of log record from the log buffer to the slave, used by the master controller when the log buffer is full and some space needs to be freed for further log records.
        Specified by:
        forceFlush in interface LogShipper
        Throws:
        java.io.IOException - If an exception occurs while trying to ship the replication message (containing the log records) across the network.
        StandardException - If an exception occurs while trying to read log records from the log buffer.
      • getHighestShippedInstant

        public long getHighestShippedInstant()
        Get the highest log instant shipped so far
        Returns:
        the highest log instant shipped so far
      • flushedInstance

        public void flushedInstance​(long latestInstanceFlushedToDisk)
        updates the information about the latest instance of the log record that has been flushed to the disk. Calling this method has no effect in this asynchronous implementation of the log shipper.
        Specified by:
        flushedInstance in interface LogShipper
        Parameters:
        latestInstanceFlushedToDisk - a long that contains the latest instance of the log record that has been flushed to the disk.
      • stopLogShipment

        public void stopLogShipment()
        Stop shipping log records. If a ship is currently in progress it will not be interrupted, shipping will stop only after the current shipment is done.
      • workToDo

        public void workToDo()
        Used to notify the log shipper that a log buffer element is full. This method would basically use the following steps to decide on the action to be taken when a notification from the log shipper is received, a) Get FI from log buffer b) If FI >= FI_HIGH b.1) notify the log shipper thread. c) Else If the time elapsed since last ship is greater than minShippingInterval c.1) notify the log shipper thread.
        Specified by:
        workToDo in interface LogShipper
      • calculateSIfromFI

        private long calculateSIfromFI()
        Will be used to calculate the shipping interval based on the fill information obtained from the log buffer. This method uses the following steps to arrive at the shipping interval, a) FI >= FI_HIGH return -1 (signifies that the waiting time should be 0) b) FI > FI_LOW and FI < FI_HIGH return minShippingInterval c) FI <= FI_LOW return maxShippingInterval.
        Returns:
        the shipping interval based on the fill information.
      • getLogShipperProperties

        private void getLogShipperProperties()
        Load relevant system properties: max and min log shipping interval