Introduction

EA & differential backup
Archive structure in brief
Scrambling
Overflow in arithmetic integer operations
Strong encryption
libdar and thread-safe requirement
Dar_manager and delete files
Native Language Support / gettext / libintl
Dar Release Process
Dar's Versions
How symmetric encryption is performed in dar/libdar
How asymmetric encryption is performed in dar/libdar

Extended attributes is a recent feature of Unix file system (at the time of the writing, year 2002). They extend attributes provided by the inode and associated to a data. They are not part of the inode, nor part of the data, nor part of a given directory. They are stored beside the inode and are a set of pair of key and value. The owner of the file can add or define any key and eventually associate data to it. It can also list and remove a particular key. What they are used for ? A way to associate information to a file.

One particular interesting use of EA, is ACL: Access Control List. ACL can be implemented using EA and add a more fine grain in assigning access permission to file. For more information one EA and ACL, see the site of Andreas Grunbacher:

to determine that an EA has changed dar looks at the ctime value. if ctime has changed, (due to EA change, but also to permission or owner change) dar saves the EA. ctime also changes, if atime or mtime changes. So if you access a file or modify it, dar will consider that the EA have changed also. This is not really fair, I admit.

Something better would be to compare EA one by one, and record those that have changed or have been deleted. But to be able to compare all EA and their value reference EA must reside in memory. As EA can grow up to 64 KB by file, this can lead to a quick saturation of the virtual memory, which is already enough solicited by the catalogue.

These two schemes implies a different pattern for saving EA in archive. In the first case (no EA in memory except at time of operation on it), to avoid skipping in the archive (and ask the user to change of disks too often), EA must be stored beside the data of the file (if present). Thus they must be distributed all along the archive (except at the end that only contains the catalogue).

In the second case (EA are loaded in memory for comparison), EA must reside beside or within the catalogue, in any case at the end of the archive, not to have to user to need all disks to just take an archive as reference.

As the catalogue, grows already fast with the number of file to save (from a few bytes for hard_link to 400 bytes around per directory inode), the memory saving option has been adopted.

Thus, EA changes are based on the ctime change. Unfortunately, no system call permits to restore ctime. Thus, restoring a differential backup after its reference has been restored, will present restored inode as more recent than those in the differential archive, thus the -r option would prevent any EA restoration. In consequence, -r has been disabled for EA, it does only concern data contents. If you don't want to restore any EA but just more recent data, you can use the following : -r -u "*"

A slice is composed of a header, data and trailer (the trailer appeared with archive format version 8)

+--------+-------------------------------------------+-------+
| header |  Data                                     |Trailer|
|        |                                           |       |
+--------+-------------------------------------------+-------+

the slice header is composed of

+-------+----------+------+-----------+-------+
| Magic | internal | flag | extension | TLV   |
| Num.  | name     | byte | byte      | list  |
+-------+----------+------+-----------+-------+

The TLV list may contain several fields:

• First slice size [type 1]
  
• Other slice size / all slice size if no first slice size is present [type 2]
  
• data_name [type 3] . This field is detailed below.
  

A TLV list is of course a list of TLV:

To know in which slice and at which position to find a particular data, dar needs to know each file's size. This is the reason why each slice contains the slice size information, in particular the last slice. In older version, dar had to read the first slice first to get this slicing information. Then it could read the archive contents at the end of the last slice. Today, reading the last slice, dar can fetch the slicing scheme from the slice header (what we just detailed) and fetch the archive contents at the end of this same last slice.

The trailer (which is one byte length) is new since archive format version 8 (released 2.4.0). It contains the value that was located in the header flag field in older archive format, telling whether the slice is the last of the archive or not. When writting down a single sliced archive (no -s option provided), both the header and the trailer tell that the slice is the last of the archive (duplicated information). However, when doing multi-sliced archive, it is not possible to known whether a slice is the last before reaching the requested amount of data per slice (which depends on the amount of byte to save, compression ratio, encryption overhead, etc.). Thus the header flag contains a value telling that to know whether the slice is the last or not, one must read the trailer.

In older format, it was necessary to seek back to update the header with the correct information when a new slice had to be created. But, keeping this behavior, it would not have been possible to make a digest "on the fly" (see --hash option). The addition of the trailer was required for that feature: to compute a md5 or sha1 hash of each slice. But, this costs one byte per slice, yes.

As seen above in the header fields, we have among others the three following identifiers:

• magic number
• internal name
• data name
  

+---------+----------------------------+-----------+--------+---------+--------+
| version |   Data                     | catalogue | term 1 | version | term 2 |
| header  |                            |           |        | trailer |        |
+---------+----------------------------+-----------+--------+---------+--------+

The version header is a short version of the trailer version. It is used when reading an archive in sequential mode, to be able to prepare the proper compression layer, and known whether escape sequence mark are present in the archive.

the version trailer (which may still be called "version header" in some part of the documentation because it was only located at the beginning of the archive in previous archive format) is composed of:

+---------+------+---------------+------+--------+--------+
| edition | algo | command line  | flag | initial|  CRC   |
|         |      |               |      | offset |        |
+---------+------+---------------+------+--------+--------+


The data is a suite of file contents, with EA if present. When tape mark is used, a copy of the CRC is placed after's file Data and file's EA, to be used when reading the archive in sequential mode. This CRC is also dropped into the catalogue which takes place at the end of the archive to be used when reading the archive in direct access mode (the default).

  ....--+---------------------+----+------------+-----------+----+---....
        |  file data          | EA | file data  | file data | EA |
        | (may be compressed) |    | (no EA)    |           |    |
  ....--+---------------------+----+------------+-----------+----+---....

the catalogue, contains all inode, directory structure and hard_links information as well as data and EA CRC. The directory structure is stored in a simple way: the inode of a directory comes, then the inode of the files it contains, then a special entry named "EOD" for End of Directory. Considering the following tree:

 - toto
    | titi
    | tutu
    | tata
    |   | blup
    |   +--
    | boum
    | coucou
    +---

it would generate the following sequence for catalogue storage:

+-------+------+------+------+------+-----+------+--------+-----+
|  toto | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
|       |      |      |      |      |     |      |        |     |
+-------+------+------+------+------+-----+------+--------+-----+

EOD takes on byte, and this way no need to store the full path of each file, just the filename is recorded.

the terminator stores the position of the beginning of the catalogue, it is the last thing to be written. Thus dar first reads the terminator, then the catalogue. Well, there is now two terminators, both are meant to be read backward. The second terminator points to the beginning of the "trailer version" which is read first in direct access mode. The first terminator points to the start of the catalogue, which is read once the adhoc compression and encryption layers has been built based on the information found on the "trailer version"

Here is an example of how data can be structured in a four slice archive:

+--------+--------+------------------------+--+
| slice  | version|  file data + EA        |Tr|
| header | header |                        |  |
+--------+--------+------------------------+--+

the first slice (just above) has been defined smaller using the -S option

+--------+-----------------------------------------------------------------+--+
| slice  |           file data + EA                                        |Tr|
| header |                                                                 |  |
+--------+-----------------------------------------------------------------+--+

+--------+-----------------------------------------------------------------+--+
| slice  |           file data + EA                                        |Tr|
| header |                                                                 |  |
+--------+-----------------------------------------------------------------+--+

+--------+---------------------+-----------+------ +---------+--------+--+
| slice  |   file data + EA    | catalogue | term 1| version | term 2 |Tr|
| header |                     |           |       | trailer |        |  |
+--------+---------------------+-----------+-------+---------+--------+--+



the last slice is smaller because there was not enough data to make it full.

The archive is written sequentially this way.

Things get a bit more complicated if we consider compression and encryption. The way the problem is addressed in dar's code is a bit like networks are designed in computer science, using the notion of layers. Here, there is a additional constraint, a given layer may or may not be present (encryption, compression, slicing for example). So all layer must have the same interface for serving the layer above them. This interface is defined by the pure virtual class "generic_file", which provides generic methods for reading, writing, skipping, getting the current offset when writing or reading data to a "generic_file". This way the compressor class acts like a file which compresses data wrote to it and writes compressed data to another "generic_file" below it. The strong encryption and scramble classes act the same but in place of compressing/uncompressing they encrypt/decrypt the data to/from another generic_file object. The slicing we have seen above follows the same principle, this is a "sar" object that transfers data wrote to it to several fichier [=file] objects. Class fichier [=file] also inherit from generic_file class, and is a wrapper for the plain file system calls. Some new classes have been added with format 8, in particular the escape class, which inserts escape sequence mark at requested position, and modifies data wrote for it never looks like an escape sequence mark. To reduce the level of context switch when reading the catalogue (which makes a ton of small read), a cache class is also present, it gather small writes made to it into larger writes, and pre-reads a large amount of data to answer to the many small reads when building the catalogue in memory from the archive.

Here are now all currently possible layers together:

              +----+--+----+-...........+---------+
archive       |file|EA|file|            |catalogue|
layout        |data|  |data|            |         |
              +----+--+----+-...........+---------+
                |   |    |      |              |
            +-----+ | +-------+ |              |
sparse      |spars| | |sparse | |              |
file        |file | | |file   | |              |
detection   |detec| | |detect.| |              |
layer       +-----+ | +-------+ |              |
(optional)      |   |    |      |              |
                V   V    V      V              V
              +-----------------------------------+
compression   |         (compressed)  data        |
              +-----------------------------------+
                    |                      |
                    |                      |
                    V                      V
              +-----------------------------------+
escape layer  |   escaped data / escape sequences |
(optional)    +-----------------------------------+
                    |                      |          / First Terminateur
                    |                      |          |
                    |                      |          V
elastic  +---+      |                      |       +----+---+
buffers  |EEE|      |                      |       | T1 |EEE|
         +---+      |                      |       +----+---+
           |        |                      |              |           Second
           V        V                      V              V         Terminator
         +--------------------------------------------------+              |
cipher   |        (encrypted) data / cache if no encryption |              |
         +--------------------------------------------------+              V
                    |                         |               +---------+----+
+-------+           |                         |               | trailer | T2 |
| header|           |                         |               +---------+----+
+-------+           |                         |                    |      |
    |               |                         |                    |      |
    V               V                         V                    V      v
+-----------------------------------------------------------------------------+
|                  data                                                       |
+-----------------------------------------------------------------------------+
        |         |  |         |   |        |   |        |  |    |  |        |
slice   |         |  |         |   |        |   |        |  |    |  |        |
headers |         |  |         |   |        |   |        |  |    |  |        |
 |  |   |         |  |         |   |        |   |        |  |    |  |        |
 |  +---|------\  |  |         |   |        |   |        |  |    |  |        |
 V      V      V  V  V         V   V        V   V        V  V    V  V        V
+---------+  +---------+  +---------+  +---------+  +-------+  +-------+  +----+
|HH| data |  |HH| data |  |HH| data |  |HH| data |  |HH|data|  |HH|data|  |HH| |
+---------+  +---------+  +---------+  +---------+  +-------+  +-------+  +----+
  slice 1      slice 2      slice 3      slice 4      slice 5




For clarity, the size field between '>' and '<' has been written in decimal instead of binary, as well as the random data inside the elastic buffer. The location of the size field '>20<' is also randomly chosen at creation time.

A Teminateur is short structure that is intended to be read backward. It gives the absolute position of a given item within the archive: The second terminateur let dar skip at the beginning of the archive trailer. The first terminateur (eventually encrypted) let dar skip at the beginning of the catalogue).

Before strong encryption was implemented, dar had only a very simple and weak encryption mechanism. It remains available in current release under the "scram" algorithm name. It mains advantage is that is does not rely on any external library, it is completely part of libdar.

Consider the pass phrase as a string, thus a sequence of bytes, thus a sequence of integer each one between 0 and 255 (including 0 and 255). The data to "scramble" is also a sequence of byte, usually much longer than the pass phrase. The principle is to add byte by byte the pass phrase to the data, modulo 256. The pass phrase is repeated all along the archive. Let's take an example:

h = 104
l = 108
o = 111
e = 101
x = 120
a = 97
m = 109
p = 112

This method allows to decode any portion without knowing the rest of the data. It does not consume much resources to compute, but it is terribly weak and easy to crack. Of course, the data is more difficult to retrieve without the key when the key is long. Today dar can also use strong encryption (blowfish and few others) and thanks to a encryption block can still avoid reading the whole archive to restore any single file.

dar relies on gpgme library (GPG made Easy) to provide strong asymmetrical encryption and signature of an archive. Asymmetrical encryption is what you do when you use a public key to cypher data (=encrypt data) and a private key to uncypher it (=decrypt it). However, dar does not encrypt the whole archive that way, not it signs it that way, neither.

Instead, dar relies on the symmetrical strong encryption algorithm it is aware of for some time (blowfish, twofish, camellia, aes, etc.) to cipher the archive. The key used to cipher the archive is chosen randomly, encrypted and eventually signed using the provided recipient email of the user keyring. This encrypted key is then put in the archive header and trailer.

Well, to be more precise about the symmetrical key that is encrypted in the archive: its length user defined and defaults to 512 bytes (4096 bits). A random variation of +0 to +255 bytes is added by libdar to this size. Then the key value itself is chosen randomly. The random generator used here is the one provided by libgcrypt using the GCRY_STRONG_RANDOM entropy level.

Why doing that way and not using the asymmetrical algorithm to cypher the whole archive?

• Because it would no more be possible to extract a single file from the archive or to read the archive's contents whithout reading the whole archive.
• Because it would not be possible the quickly verify archive signatures (well, see below)
  
• Because it would avoid recovering an corrupted archive after the point of corruption
• Because it would cost much disk space to encrypt an archive for it be readable by more than one recipients
  

Yes, you can provide several recipients from your GPG keyring giving their email addresses and also at the same time sign the archive with one of your private key. The resulting archive will be possible to decrypt only by those recipients. And anyone knowing your public key will be able to verify that the archive has been generated by you. Of course the verification only validates the encryption key is from an archive you have personnaly generated, a man in the middle could modify the archive data located after the key in the archive, however it would not be possible to uncipher the tampered data using the signed key, or it would mean that the man in the middle could generate encrypted data using the same symmetric key that is encrypted in the archive. This might be the possible if a recipient's private key has been compromised. Thus the signature of the key is not sufficient to prove the authenticity of the whole archive. To cope with that risks, a better solution is to activate the slice hashing (md5 or sha1) and to sign these small files to be provided beside the archive slices.

Some code explanation about the detection of integer arithmetic operation overflows. We speak about *unsigned* integers, and we have only portable standard ways to detect overflows when using 32 bits or 64 bits integer in place of infinint.

Developed in binary, a number is a finite suite of digits (0 or 1). To obtain the original number from the binary representation, we must multiply each digit by a power of two. example the binary representation "101101" designs the number N where:

N = 2^5 + 2^3 + 2^2 + 2^0

in that context we will say that 5 is the maximum power of N (the power of the higher non null binary digit).

for substraction "-" operation, if the second operand is greater than the first there will be an overflow (result must be unsigned thus positive) else there will not be any overflow. Thus detection is even more simple.

for division "/" and modulo "%" operations, there is never an overflow (there is just illicit the division by zero).

for multiplication "*" operation, a heuristic has been chosen to quickly detect overflow, the drawback is that it may triggers false overflow when number get near the maximum possible integer value. Here is the heuristic used:

given A and B two integers, which max powers are m and n respectively, we have

A < 2^(m+1)

and

B < 2^(n+1)

thus we also have:

A.B < 2^(m+1).2^(n+1)

which is:

A.B < 2^(m+n+2)

Several cyphers are available. Remind that "scrambling" is not a strong encryption cypher, all other are.

to be able to use a strong encrypted archive you need to know the three parameters used at creation time:

no information about these parameters is stored in the generated archive. If you make an error on just one of them, you will not be able to use your archive. If you forgot one of them, nobody can help you, you can just consider the data in this archive as lost. This is the drawback of strong encryption.

To not completely break the possibility to directly access file, the archive is  not encrypted as a whole (as would do an external program). The encryption is done block of data by block of data. Each block can be decrypted, and if you want to read some data somewhere you need to decrypt the whole block(s) it is in.

In consequence, the larger the block size is, the stronger the encryption is. But the larger the block size is too, the longer it will take to recover a given file, in particular when the file size to restore is much smaller than the encryption block size used.

An encryption block size can range from 10 bytes to 4 GB.

If encryption is used as well as compression, compression is done first, then encryption is done on compressed data.

An "elastic buffer" is introduced at the beginning and at the end of the archive, to protect against plain text attack.  The elastic buffer size randomly varies and is defined at execution time. It is composed of random (srand()) values. Two marks characters '>' and '<' delimit the size field, which indicate the byte size of the elastic buffer. The size field is randomly placed in the buffer. Last, the buffer is encrypted with the rest of the data. Typical elastic buffer size range from 1 byte to 10 kB, for both initial and terminal elastic buffers.

Elastic buffers are also used inside compression blocks. The underlying cypher may not be able to encrypt at the requested block size boundary. If necessary a small elastic buffer is appended to the data before encryption, to be able, at restoration time, to know the amount of data and the amount of noise around it.

Let's take an example with blowfish. Blowfish encrypts by multiple of 8 bytes (blowfish chain block cypher). An elastic buffer is always added to the data of a encryption block, its minimal size is 1 byte.

Thus, if you request a encryption block of 3 bytes, these 3 bytes will be padded by an elastic buffer of 5 bytes for these 8 bytes to be encrypted. This will make a very poor compression ratio as only 3 bytes on 8 bytes are significant.

If you request a encryption block of 8 bytes, as there is no room for the minimal elastic buffer of 1 byte, a second 8 byte block is used to put the elastic buffer, so the real encryption block will be 16 bytes.

Ideally, a encryption block of 7 bytes, will use 8 bytes with 1 byte for the elastic buffer.

This problem tends to disappear when the encryption block size grows, so this should not be a problem in normal conditions. Encryption block of 3 bytes is not a good idea to have a strong encryption scheme, for information, the default encryption block size is 10kB.

This is for those who plane to use libdar in their own programs.

If you plan to have only one thread using libdar there is no problem, of course, you will however have to call one of the get_version() first, as usual. Thing change if you intend to have several concurrent threads using libdar library.

libdar is thread-safe under certain conditions:

Several 'configure' options have an impact on thread-safe support:

--enable-test-memory is a debug option that avoid libdar to be thread-safe,  so don't use it.
--enable-special-alloc (set by default), makes a thread-safe library only if POSIX mutex are available (pthread_mutex_t type).
--disable-thread-safe avoid looking for mutex, so unless --disable-special-alloc is also used, the generated library will not be thread safe.

You can check the thread safe capability of a library thanks to the get_compile_time_feature(...) call from the API. Or use 'dar -V' command, to  quickly have the corresponding values and check using 'ldd' to see which library has been dynamically linked to dar, if applicable.

IMPORTANT:
As more as before it is mandatory to call get_version() call before any other call, when the call returns, libdar is ready for thread safe. Note that even if the prototype does not change get_version() *may* now throw an exception, so use get_version_noexcept() if you don't want to manage exceptions.

For more information about libdar and its API, check the doc/api_tutorial.html document and the API reference manual under doc/html/index.html

This is for further reference and explanations.

In dar archive when a file has been deleted since the backup of reference (in case of differential archive), an entry of a special type (called "detruit") is put in the catalogue of the archive which only contains the name of the missing file.

In a dar_manager database, to each files that have been found in one of the archive used to build this database corresponds a list of association. These associations put in relation the mtime (date of modification of the file) to the archive number where the file has been found in that state.

There is thus no way to record "detruit" entries in a dar_manager database, as no date is associated with this type of object. Yes, in a dar archive, we can only notice a file has been destroyed because it is not present in the filesystem but is present in the catalogue of the archive of reference. Thus we know the file has been destroyed between the date the archive of reference has been done and the date the current archive is actually done. Unfortunately, no date is recorded in dar archives telling it has been done at which time.

Thus, from dar_manager, inspecting a catalogue, there is no way to give a significant date to a "detruit" entry. In consequences, for a given file which has been removed, then recreated, then removed again along a series of differential backups, it is not possible to order the times when this file has been removed in the series of date when it has existed.

The ultimate consequence is that if the user asks dar_manager to restore a directory in the state just before a given date (-w option), it will not be possible to know if that file existed at that time. We can effectively see that it was not present in a given archive but as we don't know the date of that archive we cannot determine if it is before of after the date requested by the user, and dar_manager is not able to restore the non existence of a file for a given time, we must use dar directly with the archive that has been done at the date we wish.

Note that having a date stored in each dar archive would not solve the problem without some more informations. First, we should assume that the date is consistent from host to host and from time to time (What if the user change of time due to daylight saving or move around the Earth, or if two users in two different places share a filesystem --- with rsync, nfs, or other mean --- and do backups alternatively...). Let's assume the system time is significant and thus let's imagine what would be the matter if in each archive this date of archive construction was stored.

Then when a detruit object is met in an archive it can be given the date the archive has been built and thus ordered in the series of dates when the corresponding file was found in other archives. So when the user asks for restoration of a directory a given file's state is possible to know, and thus the restoration of the corresponding archive will do what we expect : either remove the file (if the selected backup contains an  "detruit" object, or  restore the file in the state it had).

Suppose now, a dar_manager database built with a series of full backup. There will thus no be any "detruit" objects, but a file may be present or may be missing in a given archive. The solution is thus that once an archive has been integrated in the database, the last step is to scan the whole database for files that have no date associated with this last archive, thus we can assume these files were not present and add the date of the archive creation with the information that this file was removed at that time. Moreover, if the last archive add a file which was not know in archives already present in the database, we must consider this file was deleted in each of these previous archives, but then we must record the dates of creation for all these previous archive to be able to put this information properly in the database. But, in that case we would not be able to make dar removing a file, as no "detruit" object exist (all archive are full backups), and dar_manager should remove itself the entry from the filesystem. Beside the fact that it is not the role to dar_manager to directly interact with the filesystem, dar_manager should record an additional information to know if a file is deleted because it has been found a "detruit" object in an archive, or if it is deleted because it has not been found any entry in an given archive. This is necessary to known whether to rely on dar to remove the file or to make dar_manager do it itself, or maybe better is to never rely on dar to remove a file but always let dar_manager do it itself.

Assuming we accept to make dar_manager able to rm entries from filesystem without relying on dar, we must store the date of the archive creation in each archive, and store these dates for each archive in dar_manager databases. Then instead of using the mtime of each file, we could do something much more simple in database: for each file, record if it was present or not in each archive used to built the database, and beside this, store only the archive creation date of each archive. This way, dar_manager would only have for each file to take the last state of the file (deleted or present) before the given date (or the last known state if no date is given) and either restore the file from the corresponding archive or remove it.

But if a user has removed a file by accident and only notice this mistake after several backups, it would become painful to restore this file, as the user should find manually at which date it was present to be able to feed dar_manager with the proper -w option, this worse than looking for the last archive that has the file we look for.

Here we are back to the restoration of a file and the restoration of a state. By state, I mean the state a directory tree had at a given time, like a photo. In its original version dar_manager was aimed to restore files, whatever they exist or not in the last archive added to a database. It only finds the last archive where the file is present. Making dar_manager restore a state, and thus considering files that have been removed at a given date, is no more no less than restoring from a given archive, directly with dar. So all this discussion about the fact that dar_manager is not able to handle files that have been removed, to arrive to the fact that adding this feature to dar_manager will make it become quite useless... sight. But, that was necessary.

Native Language Support (NLS) is the fact a given program can display its messages in different languages. For dar, this is implemented using the gettext tools. This tool must be installed on the system for dar can be able to display messages in another language than English. 

Things are the following:
- On a system without gettext dar will not use gettext at all. All messages will be in English (OK maybe better saying Frenglish) ;-)
- On a system with gettext dar will use the system's gettext, unless you use --disable-nls option with the configure script.

If NLS is available you just have to set the LANG environment variable to your locale settings to change the language in which dar displays its messages (see ABOUT-NLS for more about the LANG variable).

just for information, gettext() is the name of the call that makes translations of string in the program. This call is implemented in the library called 'libintl' (intl for Internationalization). Last point, gettext by translating strings, makes the Native Language Support (NLS) possible, in other words, it let you have the messages of your preferred  programs being displayed in you native language for those not having the English as mother tong.

This was necessary to say, because you may miss the links between "gettext" , "libintl" and "NLS".

READ the ABOUT-NLS file at the root of the source package to learn more about the way to display dar's messages in your own language. Note that not all languages are yet supported, this is up to you to send me a translation in your language and/or contact a translating team as explained in ABOUT-NLS.

To know which languages are supported by dar, read the po/LINGUAS file and check out for the presence of the corresponding *.po files in this directory.