Class BaseDataFileFactory

  • All Implemented Interfaces:
    java.security.PrivilegedExceptionAction<java.lang.Object>, CacheableFactory, ModuleControl, ModuleSupportable, Corruptable, DataFactory
    Direct Known Subclasses:
    BaseDataFileFactoryJ4

    public class BaseDataFileFactory
    extends java.lang.Object
    implements DataFactory, CacheableFactory, ModuleControl, ModuleSupportable, java.security.PrivilegedExceptionAction<java.lang.Object>
    Provides the abstract class with most of the implementation of DataFactory and ModuleControl shared by all the different filesystem implementations.

    RESOLVE (mikem - 2/19/98) - Currently only getContainerClass() is abstract, there are probably more routines which should be abstract. Also the other implementations should probably inherit from the abstract class, rather than from the DataFileFactory class. Also there probably should be a generic directory and the rest of the filesystem implementations parallel to it. I wanted to limit the changes going into the branch and then fix inheritance stuff in main.

    The code in this class was moved over from DataFileFactory.java and then that file was made to inherit from this one.

    • Field Detail

      • nextContainerId

        private long nextContainerId
      • databaseEncrypted

        private boolean databaseEncrypted
      • jvmVersion

        private java.lang.String jvmVersion
      • osInfo

        private java.lang.String osInfo
      • jarCPath

        private java.lang.String jarCPath
      • dataDirectory

        private java.lang.String dataDirectory
      • throwDBlckException

        private boolean throwDBlckException
      • identifier

        private UUID identifier
      • freezeSemaphore

        private final java.lang.Object freezeSemaphore
      • isFrozen

        private boolean isFrozen
      • writersInProgress

        private int writersInProgress
      • removeStubsOK

        private boolean removeStubsOK
      • isCorrupt

        private boolean isCorrupt
      • inCreateNoLog

        private boolean inCreateNoLog
      • dataNotSyncedAtAllocation

        boolean dataNotSyncedAtAllocation
      • dataNotSyncedAtCheckpoint

        boolean dataNotSyncedAtCheckpoint
      • loggablePageActions

        private PageActions loggablePageActions
      • readOnly

        private boolean readOnly
      • supportsRandomAccess

        private boolean supportsRandomAccess
      • droppedTableStubInfo

        private java.util.Hashtable<LogInstant,​java.lang.Object[]> droppedTableStubInfo
      • postRecoveryRemovedFiles

        private java.util.Hashtable<java.lang.String,​StorageFile> postRecoveryRemovedFiles
      • actionCode

        private int actionCode
      • REMOVE_TEMP_DIRECTORY_ACTION

        private static final int REMOVE_TEMP_DIRECTORY_ACTION
        See Also:
        Constant Field Values
      • GET_CONTAINER_PATH_ACTION

        private static final int GET_CONTAINER_PATH_ACTION
        See Also:
        Constant Field Values
      • GET_ALTERNATE_CONTAINER_PATH_ACTION

        private static final int GET_ALTERNATE_CONTAINER_PATH_ACTION
        See Also:
        Constant Field Values
      • FIND_MAX_CONTAINER_ID_ACTION

        private static final int FIND_MAX_CONTAINER_ID_ACTION
        See Also:
        Constant Field Values
      • DELETE_IF_EXISTS_ACTION

        private static final int DELETE_IF_EXISTS_ACTION
        See Also:
        Constant Field Values
      • POST_RECOVERY_REMOVE_ACTION

        private static final int POST_RECOVERY_REMOVE_ACTION
        See Also:
        Constant Field Values
      • RELEASE_LOCK_ON_DB_ACTION

        private static final int RELEASE_LOCK_ON_DB_ACTION
        See Also:
        Constant Field Values
      • RESTORE_DATA_DIRECTORY_ACTION

        private static final int RESTORE_DATA_DIRECTORY_ACTION
        See Also:
        Constant Field Values
      • GET_CONTAINER_NAMES_ACTION

        private static final int GET_CONTAINER_NAMES_ACTION
        See Also:
        Constant Field Values
      • stub

        private boolean stub
      • myUUID

        private UUID myUUID
      • databaseDirectory

        private java.lang.String databaseDirectory
      • backupRoot

        private java.io.File backupRoot
      • bfilelist

        private java.lang.String[] bfilelist
    • Constructor Detail

      • BaseDataFileFactory

        public BaseDataFileFactory()
    • Method Detail

      • canSupport

        public boolean canSupport​(java.util.Properties startParams)
        Description copied from interface: ModuleSupportable
        See if this implementation can support any attributes that are listed in properties. This call may be made on a newly created instance before the boot() method has been called, or after the boot method has been called for a running module.

        The module can check for attributes in the properties to see if it can fulfill the required behaviour. E.g. the raw store may define an attribute called RawStore.Recoverable. If a temporary raw store is required the property RawStore.recoverable=false would be added to the properties before calling bootServiceModule. If a raw store cannot support this attribute its canSupport method would return null. Also see the Monitor class's prologue to see how the identifier is used in looking up properties.
        Actually a better way maybe to have properties of the form RawStore.Attributes.mandatory=recoverable,smallfootprint and RawStore.Attributes.requested=oltp,fast

        Specified by:
        canSupport in interface ModuleSupportable
        Returns:
        true if this instance can be used, false otherwise.
      • boot

        public void boot​(boolean create,
                         java.util.Properties startParams)
                  throws StandardException
        Description copied from interface: ModuleControl
        Boot this module with the given properties. Creates a module instance that can be found using the findModule() methods of Monitor. The module can only be found using one of these findModule() methods once this method has returned.

        An implementation's boot method can throw StandardException. If it is thrown the module is not registered by the monitor and therefore cannot be found through a findModule(). In this case the module's stop() method is not called, thus throwing this exception must free up any resources.

        When create is true the contents of the properties object will be written to the service.properties of the persistent service. Thus any code that requires an entry in service.properties must explicitly place the value in this properties set using the put method.
        Typically the properties object contains one or more default properties sets, which are not written out to service.properties. These default sets are how callers modify the create process. In a JDBC connection database create the first set of defaults is a properties object that contains the attributes that were set on the jdbc:derby: URL. This attributes properties set has the second default properties set as its default. This set (which could be null) contains the properties that the user set on their DriverManager.getConnection() call, and are thus not owned by Derby code, and thus must not be modified by Derby code.

        When create is false the properties object contains all the properties set in the service.properties file plus a limited number of attributes from the JDBC URL attributes or connection properties set. This avoids properties set by the user compromising the boot process. An example of a property passed in from the JDBC world is the bootPassword for encrypted databases.

        Code should not hold onto the passed in properties reference after boot time as its contents may change underneath it. At least after the complete boot is completed, the links to all the default sets will be removed.

        Specified by:
        boot in interface ModuleControl
        Throws:
        StandardException - Module cannot be started.
        See Also:
        Monitor, ModuleFactory
      • stop

        public void stop()
        Description copied from interface: ModuleControl
        Stop the module. The module may be found via a findModule() method until some time after this method returns. Therefore the factory must be prepared to reject requests to it once it has been stopped. In addition other modules may cache a reference to the module and make requests of it after it has been stopped, these requests should be rejected as well.
        Specified by:
        stop in interface DataFactory
        Specified by:
        stop in interface ModuleControl
        See Also:
        Monitor, ModuleFactory
      • openContainer

        public ContainerHandle openContainer​(RawTransaction t,
                                             ContainerKey containerId,
                                             LockingPolicy locking,
                                             int mode)
                                      throws StandardException
        Description copied from interface: DataFactory
        Open a container that is not droped.
        Specified by:
        openContainer in interface DataFactory
        Parameters:
        t - the raw transaction that is opening the container
        containerId - the container's identity
        locking - the locking policy
        mode - see the different mode in @see ContainerHandle then will return a null handle if the container is dropped.
        Returns:
        the handle to the opened container
        Throws:
        StandardException - Standard Derby error policy
      • addContainer

        public long addContainer​(RawTransaction t,
                                 long segmentId,
                                 long input_containerid,
                                 int mode,
                                 java.util.Properties tableProperties,
                                 int temporaryFlag)
                          throws StandardException
        Add a container with a specified page size to a segment.
        Specified by:
        addContainer in interface DataFactory
        Parameters:
        t - the transaction that is creating the container
        segmentId - the segment where the container is to go
        mode - whether or not to LOGGED or not. The effect of this mode is only for this addContainer call, not persisently stored throughout the lifetime of the container
        tableProperties - properties of the container that is persistently stored throughout the lifetime of the container
        Returns:
        the containerId of the newly created container
        Throws:
        StandardException - Standard Derby error policy
      • addAndLoadStreamContainer

        public long addAndLoadStreamContainer​(RawTransaction t,
                                              long segmentId,
                                              java.util.Properties tableProperties,
                                              RowSource rowSource)
                                       throws StandardException
        Add and load a stream container
        Specified by:
        addAndLoadStreamContainer in interface DataFactory
        Parameters:
        t - the transaction that is creating the container
        segmentId - the segment where the container is to go
        tableProperties - properties of the container that is persistently stored throughout the lifetime of the container
        rowSource - the data to load the container with
        Returns:
        the containerId of the newly created stream container
        Throws:
        StandardException - Standard Derby error policy
      • dropContainer

        public void dropContainer​(RawTransaction t,
                                  ContainerKey ckey)
                           throws StandardException
        Drop a container.

        Synchronisation

        This call will mark the container as dropped and then obtain an CX lock (table level exclusive lock) on the container. Once a container has been marked as dropped it cannot be retrieved by an openContainer() call unless explicitly with droppedOK.

        Once the exclusive lock has been obtained the container is removed and all its pages deallocated. The container will be fully removed at the commit time of the transaction.

        Specified by:
        dropContainer in interface DataFactory
        Throws:
        StandardException - Standard Derby error policy
      • checkpoint

        public void checkpoint()
                        throws StandardException
        Implement checkpoint operation, write/sync all pages in cache.

        The derby write ahead log algorithm uses checkpoint of the data cache to determine points of the log no longer required by restart recovery.

        This implementation uses the 2 cache interfaces to force all dirty pages to disk: WRITE DIRTY PAGES TO OS: In the first step all pages in the page cache are written, but not synced (pagecache.cleanAll). The cachemanager cleanAll() interface guarantees that every dirty page that exists when this call is first made will have it's clean() method called. The data cache (CachedPage.clean()), will call writePage but not sync the page. By using the java write then sync, the checkpoint is usually doing async I/O, allowing the OS to schedule multiple I/O's to the file as efficiently as it can. Note that it has been observed that checkpoints can flood the I/O system because these writes are not synced, see DERBY-799 - checkpoint should probably somehow restrict the rate it sends out those I/O's - it was observed a simple sleep every N writes fixed most of the problem. FORCE THOSE DIRTY WRITES TO DISK: To force the I/O's to disk, the system calls each open dirty file and uses the java interface to sync any outstanding dirty pages to disk (containerCache.cleanAll()). The open container cache does this work in RAFContainer.clean() by writing it's header out and syncing the file. (Note if any change is made to checkpoint to sync the writes vs. syncing the file, one probably still needs to write the container header out and sync it).

        Specified by:
        checkpoint in interface DataFactory
        Throws:
        StandardException - Standard exception policy.
      • markCorrupt

        public StandardException markCorrupt​(StandardException originalError)
        Really this is just a convience routine for callers that might not have access to a log factory.
        Specified by:
        markCorrupt in interface Corruptable
        Returns:
        Must always return its parameter.
      • removeStubsOK

        public void removeStubsOK()
        Description copied from interface: DataFactory
        Tell the data factory it is OK to remove committed deleted containers when the data factory shuts down.
        Specified by:
        removeStubsOK in interface DataFactory
      • setUndoInsertEventHandler

        public void setUndoInsertEventHandler​(UndoHandler input_undo_handle)
        Register a handler class for insert undo events.

        Register a class to be called when an undo of an insert is executed. When an undo of an event is executed by the raw store UndoHandler.insertUndoNotify() will be called, allowing upper level callers to execute code as necessary. The initial need is for the access layer to be able to queue post commit reclaim space in the case of inserts which are aborted (including the normal case of inserts failed for duplicate key violations) (see DERBY-4057)

        Specified by:
        setUndoInsertEventHandler in interface DataFactory
        Parameters:
        input_undo_handle - Class to use to notify callers of an undo of an insert.
        Throws:
        StandardException - Standard exception policy.
      • insertUndoNotify

        protected void insertUndoNotify​(RawTransaction rxact,
                                        PageKey page_key)
                                 throws StandardException
        Notify through set handler that an undo of an insert has happened.

        When an undo of an event is executed by the raw store UndoHandler.insertUndoNotify() will be called, allowing upper level callers to execute code as necessary. The initial need is for the access layer to be able to queue post commit reclaim space in the case of inserts which are aborted (including the normal case of inserts failed for duplicate key violations) (see DERBY-4057) Longer descrption of routine.

        Parameters:
        rxact - raw transaction of the aborted insert.
        page_key - page key of the aborted insert.
        Throws:
        StandardException - Standard exception policy.
      • getIntParameter

        public int getIntParameter​(java.lang.String parameterName,
                                   java.util.Properties properties,
                                   int defaultValue,
                                   int minimumValue,
                                   int maximumValue)
      • getRootDirectory

        public java.lang.String getRootDirectory()
        Description copied from interface: DataFactory

        Get the root directory of the data storage area. It is always guaranteed to be an absolute path, and it is prefixed with the JDBC sub-sub-protocol if it is not a directory database. Examples:

        /path/to/database
        in case of a directory database
        memory:/path/to/database
        in case of a memory database
        Specified by:
        getRootDirectory in interface DataFactory
        Returns:
        the root directory of the data storage area
      • newContainerObject

        Cacheable newContainerObject()
        Produces new container objects.

        Concrete implementations of a DataFactory must implement this routine to indicate what kind of containers are produced. This class produces file-based containers - RAFContainer objects for files that support random access and InputStreamContainer object for others, such as data files in JARs.

        Returns:
        A new file container object.
      • newRAFContainer

        protected Cacheable newRAFContainer​(BaseDataFileFactory factory)
        Creates a RAFContainer object. This method is overridden in BaseDataFileFactoryJ4 to produce RAFContainer4 objects instead of RAFContainer objects.
      • getLoggableAllocationActions

        private AllocationActions getLoggableAllocationActions()
        Get the loggable allocation action associated with this implementation
        Returns:
        the PageActions
      • removeTempDirectory

        private void removeTempDirectory()
      • getContainerPath

        public StorageFile getContainerPath​(ContainerKey containerId,
                                            boolean stub)
        Return the path to a container file.

        Return the path to a container file that is relative to the root directory.

        The format of the name of an existing container file is: segNNN/cXXX.dat The format of the name of a stub describing a dropped container file is: segNNN/dXXX.dat NNN = segment number, currently 0 is where normal db files are found. XXX = The hex representation of the container number The store will always create containers with this format name, but the store will also recognize the following two formats when attempting to open files - as some copy tools have uppercased our filesnames when moving across operating systems: The format of the name of an existing container file is: segNNN/CXXX.DAT The format of the name of a stub describing a dropped container file is: segNNN/DXXX.DAT

        Parameters:
        containerId - The container being opened/created
        stub - True if the file name for the stub is requested, otherwise the file name for the data file
        Returns:
        The StorageFile representing path to container relative to root.
      • getAlternateContainerPath

        public StorageFile getAlternateContainerPath​(ContainerKey containerId,
                                                     boolean stub)
        Return an alternate path to container file relative to the root directory. The alternate path uses upper case 'C','D', and 'DAT' instead of lower case - there have been cases of people copying the database and somehow upper casing all the file names. The intended use is as a bug fix for track 3444.
        Parameters:
        containerId - The container being opened/created
        stub - True if the file name for the stub is requested, otherwise the file name for the data file
      • removeStubs

        private void removeStubs()
        Remove stubs in this database. Stubs are committed deleted containers
      • stubFileToRemoveAfterCheckPoint

        public void stubFileToRemoveAfterCheckPoint​(StorageFile file,
                                                    LogInstant logInstant,
                                                    java.lang.Object identity)
        keeps track of information about the stub files of the committed deleted containers. We use the info to delete them at checkpoints. In addition to the file info , we also keep track of the identity of the container; which helps to remove entry in the cache and the log instant when the stub was created, which helps us to figure out whether we require the stub file for the crash recovery. We maintain the information in a hashtable: key(LOG INSTANT) Values: File handle , and ContainerIdentity.
      • removeDroppedContainerFileStubs

        public void removeDroppedContainerFileStubs​(LogInstant redoLWM)
                                             throws StandardException
        Delete the stub files that are not required for recovery. A stub file is not required to be around if the recovery is not going to see any log record that belongs to that container. Since the stub files are created as a post commit operation, they are not necessary during undo operation of the recovery. To remove a stub file we have to be sure that it was created before the redoLWM in the check point record. We can be sure that the stub is not required if the log instant when it was created is less than the redoLWM.
        Specified by:
        removeDroppedContainerFileStubs in interface DataFactory
        Throws:
        StandardException - Standard Derby error policy
      • findMaxContainerId

        private long findMaxContainerId()
        Find the largest containerid is seg 0.

        Do a file list of the files in seg0 and return the highest numbered file found.

        Until I figure out some reliable place to store this information across a boot of the system, this is what is used following a boot to assign the next conglomerate id when a new conglomerate is created. It is only called at most once, and then the value is cached by calling store code.

        Returns:
        The largest containerid in seg0.
      • handleServiceType

        private boolean handleServiceType​(java.lang.String type)
        Does this factory support this service type.
      • getJBMSLockOnDB

        private void getJBMSLockOnDB​(UUID myUUID,
                                     UUIDFactory uuidFactory,
                                     java.lang.String databaseDirectory)
                              throws StandardException
        check to see if we are the only JBMS opened against this database.
        This method does nothing if this database is read only or we cannot access files directly on the database directory.
        We first see if a file named db.lck exists on the top database directory (i.e., the directory where service.properties lives). If it doesn't exist, we create it and write to it our identity which is generated per boot of the JBMS.
        If the db.lck file already exists when we boot this database, we try to delete it first, assuming that an opened RandomAccessFile can act as a file lock against delete. If that succeeds, we may hold a file lock against subsequent JBMS that tries to attach to this database before we exit.
        We test to see if we think an opened file will prevent it from being deleted, if so, we will hold on to the open file descriptor and use it as a filelock. If not, and we started out deleting an existing db.lck file, we issue a warning message to the info stream that we are about to attached to a database which may already have another JBMS attached to it. Then we overwrite that db.lck file with our identity.
        Upon shutdown, we delete the db.lck file. If the system crash instead of shutdown cleanly, it will be cleaned up the next time the system boots
        Throws:
        StandardException - another JBMS is already attached to the database at this directory
      • releaseJBMSLockOnDB

        private void releaseJBMSLockOnDB()
      • privReleaseJBMSLockOnDB

        private void privReleaseJBMSLockOnDB()
                                      throws java.io.IOException
        Throws:
        java.io.IOException
      • logMsg

        private void logMsg​(java.lang.String msg)
      • databaseEncrypted

        public final boolean databaseEncrypted()
        Description copied from interface: DataFactory
        Returns if data base is in encrypted mode.
        Specified by:
        databaseEncrypted in interface DataFactory
        Returns:
        true if database encrypted false otherwise
      • setDatabaseEncrypted

        public void setDatabaseEncrypted​(boolean isEncrypted)
        Sets whether the database is encrypted.
        Specified by:
        setDatabaseEncrypted in interface DataFactory
        Parameters:
        isEncrypted - true if the database is encrypted, false otherwise
      • jarClassPath

        private static java.lang.String jarClassPath​(java.lang.Class cls)
        Return a jar file by asking the class's class loader for the location where the class was loaded from. If no value, it returns null
        Parameters:
        cls - the Class to ask to print the class name of an object
        Returns:
        the ClassPath of a jar file
      • buildOSinfo

        private static java.lang.String buildOSinfo()
        Return values of system properties that identify the OS. Will catch SecurityExceptions and note them for displaying information.
        Returns:
        the Java system property value for the OS or a string capturing a security exception.
      • buildJvmVersion

        private static java.lang.String buildJvmVersion()
        Return values of system properties that identify the JVM. Will catch SecurityExceptions and note them for displaying information.
        Returns:
        the Java system property value from the JVM or a string capturing a security exception.
      • getEncryptionBlockSize

        public int getEncryptionBlockSize()
        Returns the encryption block size used by the algorithm at time of creation of an encrypted database
        Specified by:
        getEncryptionBlockSize in interface DataFactory
      • getVersionedName

        public java.lang.String getVersionedName​(java.lang.String name,
                                                 long generationId)
      • getMaxContainerId

        public long getMaxContainerId()
                               throws StandardException
        Return an id which can be used to create a container.

        Return an id number with is greater than any existing container in the current database. Caller will use this to allocate future container numbers - most likely caching the value and then incrementing it as it is used.

        Specified by:
        getMaxContainerId in interface DataFactory
        Returns:
        The an id which can be used to create a container.
        Throws:
        StandardException - Standard exception policy.
      • getNextId

        long getNextId()
      • random

        int random()
        return a secure random number
      • fileToRemove

        void fileToRemove​(StorageFile file,
                          boolean remove)
        Add a file to the list of files to be removed post recovery.
      • setupCacheCleaner

        public void setupCacheCleaner​(DaemonService daemon)
        Set up the cache cleaner for the container cache and the page cache.
        Specified by:
        setupCacheCleaner in interface DataFactory
        Parameters:
        daemon - daemon service to use for background cleaning
      • unfreezePersistentStore

        public void unfreezePersistentStore()
        Description copied from interface: DataFactory
        Backup restore - start writing dirty pages or container to disk
        Specified by:
        unfreezePersistentStore in interface DataFactory
      • writeInProgress

        public void writeInProgress()
                             throws StandardException
        Description copied from interface: DataFactory
        Backup restore - don't allow the persistent store to be frozen - or if it is already frozen, block. A write is about to commence.
        Specified by:
        writeInProgress in interface DataFactory
        Throws:
        StandardException - Standard Derby error policy
      • writeFinished

        public void writeFinished()
        Description copied from interface: DataFactory
        Backup restore - write finished, if this is the last writer, allow the persistent store to proceed.
        Specified by:
        writeFinished in interface DataFactory
      • getContainerNames

        java.lang.String[] getContainerNames()
        get all the names of the files in seg 0. MT - This method needs to be synchronized to avoid conflicts with other privileged actions execution in this class.
        Returns:
        An array of all the file names in seg0.
      • restoreDataDirectory

        private void restoreDataDirectory​(java.lang.String backupPath)
                                   throws StandardException
        removes the data directory(seg*) from database home directory and restores it from backup location. This function gets called only when any of the folling attributes are specified on connection URL: Attribute.CREATE_FROM (Create database from backup if it does not exist) Attribute.RESTORE_FROM (Delete the whole database if it exists and then restore * it from backup) Attribute.ROLL_FORWARD_RECOVERY_FROM:(Perform Rollforward Recovery; except for the log directory everthing else is replced by the copy from backup. log files in the backup are copied to the existing online log directory. In all the cases, data directory(seg*) is replaced by the data directory directory from backup when this function is called.
        Throws:
        StandardException
      • isReadOnly

        public boolean isReadOnly()
        Is the store read-only.
        Specified by:
        isReadOnly in interface DataFactory
      • run

        public final java.lang.Object run()
                                   throws java.io.IOException,
                                          StandardException
        Specified by:
        run in interface java.security.PrivilegedExceptionAction<java.lang.Object>
        Throws:
        java.io.IOException
        StandardException
      • getMonitor

        private static ModuleFactory getMonitor()
        Privileged Monitor lookup. Must be private so that user code can't call this entry point.
      • startSystemModule

        private static java.lang.Object startSystemModule​(java.lang.String factoryInterface)
                                                   throws StandardException
        Privileged startup. Must be private so that user code can't call this entry point.
        Throws:
        StandardException
      • bootServiceModule

        private static java.lang.Object bootServiceModule​(boolean create,
                                                          java.lang.Object serviceModule,
                                                          java.lang.String factoryInterface,
                                                          java.util.Properties properties)
                                                   throws StandardException
        Privileged startup. Must be private so that user code can't call this entry point.
        Throws:
        StandardException