Compresses data into archives, supporting a variety of compression formats.
The Compression class provides an interface to compress and decompress data. It provides support for file based compression as well as memory based compression routines. The base class uses zip algorithms to support pkzip files, while other forms of compressed data can be supported by installing additional compression sub-classes.
The following examples demonstrate basic usage of compression objects in Fluid:
// Create a new zip archive and compress two files. cmp = obj.new('compression', { path='temp:result.zip', flags='!NEW' } ) err = cmp.mtCompressFile('config:defs/compression.def', '') err = cmp.mtCompressFile('config:defs/core.def', '') // Decompress all *.def files in the root of an archive. cmp = obj.new('compression', { path='temp:result.zip' } ) err = cmp.mtDecompressFile('*.def', 'temp:')
It is strongly advised that Compression objects are created for the purpose of either writing to, or reading from the target archive. The class is not designed for both purposes simultaneously, particularly due to considerations for maximising operational speed.
If data is to be encrypted or decrypted, set the Password field with a null-terminated encryption string. If the password for an encrypted file, errors will be returned when trying to decompress the information (the source archive may be reported as a corrupted file).
To list the contents of an archive, use the Scan() method.
To adjust the level of compression used to pack each file, set the CompressionLevel field to a value between 0 and 100%.
This code is based on the work of Jean-loup Gailly and Mark Adler.
The Compression class consists of the following fields:
Access | Name | Type | Comment | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ArchiveName | STRING | Apply an archive name to the object, allowing it to be used as a named object in the file system. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Setting the ArchiveName will allow a Compression object's files to be accessible using standard file system paths. This is achieved through use of the | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CompressionLevel | INT | The compression level to use when compressing data. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The level of compression that is used when compressing data is determined by the value in this field. Values range between 0 for no compression and 100 for maximum compression. The speed of compression decreases with higher values, but the compression ratio will improve. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Feedback | FUNCTION | Provides feedback during the de/compression process. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
To receive feedback during any de/compression process, set a callback routine in this field. The format for the callback routine is For object classes, the object that initiated the de/compression process can be learned by calling the Core's CurrentContext() function. During the processing of multiple files, any individual file can be skipped by returning The CompressionFeedback structure consists of the following fields:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Flags | CMF | Optional flags. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MinOutputSize | INT | Indicates the minimum output buffer size that will be needed during de/compression. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This field indicates the minimum allowable buffer size for output during decompression and compression processing. This field must be checked before allocating your own buffers for holding compressed and decompressed output, as failing to allocate enough buffer space is extremely likely to result in overflow errors. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Output | OBJECTID | Resulting messages will be sent to the object referred to in this field. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
If this field is set to a valid object ID, text messages will be sent to that object when the compression object is used. This can be helpful for notifying the user of the results of compression, decompression and removal of files. The target object must be capable of processing incoming text from data channels. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Password | STRING | Required if an archive needs an encryption password for access. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Set the Password field if an archive will use a password for the encryption of its contents. The string must be null-terminated and not more than 128 bytes in length. It is recommended that the Password is set before or immediately after initialisation. To change the password of an existing archive, create a new compression object with the desired password and transfer the existing data across to it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Path | STRING | Set if the compressed data originates from, or is to be saved to a file source. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
To load or create a new file archive, set the Path field to the path of that file. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Permissions | PERMIT | Default permissions for decompressed files are defined here. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
By default the permissions of files added to an archive are derived from their source location. This behaviour can be over-ridden by setting the Permissions field. Valid permission flags are outlined in the File class.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Size | BIGINT | Indicates the size of the source archive, in bytes. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TotalOutput | BIGINT | The total number of bytes that have been output during the compression or decompression of streamed data. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
UncompressedSize | BIGINT | The total decompressed size of all files in an archive. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
If you would like to know the total number of bytes that have been compressed into a compression object, read this field. This will tell you the maximum byte count used if every file were to be decompressed. Header and tail information that may identify the compressed data is not included in the total. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
WindowBits | INT | Special option for certain compression formats. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The WindowBits field defines the size of the sliding window frame for the default compression format (DEFLATE), but may also be of use to other compression formats that use this technique. For DEFLATE compression, the window bits range must lie between 8 and 15. Please note that if the value is negative, the algorithm will not output the traditional zlib header information. To support GZIP decompression, please set the WindowBits value to 47. |
The following actions are currently supported:
Name | Comment | |
---|---|---|
Flush | Flushes all pending actions. |
The following methods are currently supported:
Name | Comment | ||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CompressBuffer | Compresses a plain memory area into an empty buffer. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::CompressBuffer(OBJECTPTR Object, APTR Input, LONG InputSize, APTR Output, LONG OutputSize, LONG * Result)
This method provides a simple way of compressing a memory area into a buffer. It requires a reference to the source data and a buffer large enough to accept the compressed information. Generally the destination buffer should be no smaller than 75% of the size of the source data. If the destination buffer is not large enough, an error of To decompress the data that is output by this function, use the DecompressBuffer() method. The compression method used to compress the data will be identified in the first 32 bits of output, for example, Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
CompressFile | Add files to a compression object. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::CompressFile(OBJECTPTR Object, CSTRING Location, CSTRING Path)
The CompressFile method is used to add new files and folders to a compression object. The client must supply the To compress all contents of a folder, specify its path in the The Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
CompressStream | Compresses streamed data into a buffer. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::CompressStream(OBJECTPTR Object, APTR Input, LONG Length, FUNCTION * Callback, APTR Output, LONG OutputSize)
Use the CompressStream method to compress incoming streams of data whilst using a minimal amount of memory. The compression process is handled in three phases of Start, Compress and End. The methods provided for each phase are CompressStreamStart(), CompressStream() and CompressStreamEnd(). A compression object can manage only one compression stream at any given time. If it is necessary to compress multiple streams at once, create a compression object for each individual stream. No meta-information is written to the stream, so the client will need a way to record the total number of bytes that have been output during the compression process. This value must be stored somewhere in order to decompress the stream correctly. There is also no header information recorded to identify the type of algorithm used to compress the stream. We recommend that the compression object's sub-class ID is stored for future reference. The following C code illustrates a simple means of compressing a file to another file using a stream: if (auto error = mtCompressStreamStart(compress); error IS ERR::Okay) { LONG len; LONG cmpsize = 0; UBYTE input[4096]; while ((error = acRead(file, input, sizeof(input), &len)) IS ERR::Okay) { if (!len) break; // No more data to read. error = mtCompressStream(compress, input, len, &callback, NULL, 0); if (error != ERR::Okay) break; if (result > 0) { cmpsize += result; error = acWrite(outfile, output, result, &len); if (error != ERR::Okay) break; } } if (error IS ERR::Okay) { if ((error = mtCompressStreamEnd(compress, NULL, 0)) IS ERR::Okay) { cmpsize += result; error = acWrite(outfile, output, result, &len); } } } Please note that, depending on the type of algorithm, this method will not always write data to the output buffer. The algorithm may store a copy of the input and wait for more data for efficiency reasons. Any unwritten data will be resolved when the stream is terminated with CompressStreamEnd(). To check if data was output by this function, either set a flag in the callback function or compare the TotalOutput value to its original setting before CompressStream was called. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
CompressStreamEnd | Ends the compression of an open stream. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::CompressStreamEnd(OBJECTPTR Object, FUNCTION * Callback, APTR Output, LONG OutputSize)
To end the compression process, this method must be called to write any final blocks of data and remove the resources that were allocated. The expected format of the Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
CompressStreamStart | Initialises a new compression stream. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::CompressStreamStart(OBJECTPTR Object) The level of compression is determined by the CompressionLevel field value. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressBuffer | Decompresses data originating from the CompressBuffer() method. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressBuffer(OBJECTPTR Object, APTR Input, APTR Output, LONG OutputSize, LONG * Result)
This method is used to decompress data that has been packed using the CompressBuffer() method. A pointer to the compressed data and an output buffer large enough to contain the decompressed data are required. If the output buffer is not large enough to contain the data, the method will write out as much information as it can and then return with an error code of Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressFile | Extracts one or more files from a compression object. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressFile(OBJECTPTR Object, CSTRING Path, CSTRING Dest, LONG Flags)
Use the DecompressFile() method to decompress a file or files to a destination folder. The exact path name of the compressed file is required for extraction unless using wildcards. A single asterisk in the Path parameter will extract all files in a compression object. When specifying the This method sends feedback at regular intervals during decompression. For further information on receiving feedback, please refer to the Feedback field. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressObject | Decompresses one file to a target object. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressObject(OBJECTPTR Object, CSTRING Path, OBJECTPTR Object)
The DecompressObject method will decompress a file to a target object, using a series of Write() calls. This method sends feedback at regular intervals during decompression. For further information on receiving feedback, please refer to the Feedback field. Note that if decompressing to a File object, the seek position will point to the end of the file after this method returns. Reset the seek position to zero if the decompressed data needs to be read back. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressStream | Decompresses streamed data to an output buffer. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressStream(OBJECTPTR Object, APTR Input, LONG Length, FUNCTION * Callback, APTR Output, LONG OutputSize)
Call DecompressStream repeatedly to decompress a data stream and process the results in a callback routine. The client will need to provide a pointer to the data in the The format of the The Optionally, the client can specify an output buffer in the When there is no more data in the decompression stream or if an error has occurred, the client must call DecompressStreamEnd(). Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressStreamEnd | Must be called at the end of the decompression process. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressStreamEnd(OBJECTPTR Object, FUNCTION * Callback)
To end the decompression process, this method must be called to write any final blocks of data and remove the resources that were allocated during decompression. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
DecompressStreamStart | Initialises a new decompression stream. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::DecompressStreamStart(OBJECTPTR Object) Use the DecompressStreamStart method to initialise a new decompression stream. No parameters are required. If a decompression stream is already active at the time that this method is called, all resources associated with that stream will be deallocated so that the new stream can be initiated. To decompress the data stream, follow this call with repeated calls to DecompressStream() until all the data has been processed, then call DecompressStreamEnd(). Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Find | Find the first item that matches a given filter. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::Find(OBJECTPTR Object, CSTRING Path, LONG CaseSensitive, LONG Wildcard, struct CompressedItem ** Item)
Use the Find() method to search for a specific item in an archive. The algorithm will return the first item that matches the If successful, the discovered item is returned as a !CompressedItem. The result is temporary and values will be discarded on the next call to this method. If persistent values are required, copy the resulting structure immediately after the call. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
RemoveFile | Deletes one or more files from a compression object. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::RemoveFile(OBJECTPTR Object, CSTRING Path)
This method deletes compressed files from a compression object. If the file is in a folder then the client must specify the complete path in conjunction with the file name. Wild cards are accepted if you want to delete multiple files. A Depending on internal optimisation techniques, the compressed file may not shrink from deletions until the compression object is closed or the Flush() action is called. Error Codes
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Scan | Scan the archive's index of compressed data. | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERR cmp::Scan(OBJECTPTR Object, CSTRING Folder, CSTRING Filter, FUNCTION * Callback)
Use the Scan() method to search an archive's list of items. Optional filtering can be applied using the The CompressedItem structure consists of the following fields:
To search for a single item with a path and name already known, use the Find() method instead. Error Codes
|
Compression flags
Name | Description |
---|---|
CMF::APPLY_SECURITY | When decompressing, apply individual file permissions if they are available in the compression file. |
CMF::CREATE_FILE | Create a new archive only if the source file does not already exist. |
CMF::NEW | Force the creation of a new file archive. Any existing file data at the target location will be destroyed. |
CMF::NO_LINKS | Treat symbolic links as normal files/folders. |
CMF::PASSWORD | A password has been set on the object. |
CMF::READ_ONLY | Forces read-only access, which is strongly recommended if an existing archive is being opened with no modifications intended. If this flag is not set, initialisation can fail if the user does not have write access to the source file. |
Permission flags
Name | Description |
---|---|
PERMIT::ALL_DELETE | Synonym for EVERYONE_DELETE |
PERMIT::ALL_EXEC | Synonym for EVERYONE_EXEC |
PERMIT::ALL_READ | Synonym for EVERYONE_READ |
PERMIT::ALL_WRITE | Synonym for EVERYONE_WRITE |
PERMIT::ARCHIVE | Marks the file for future backup. The flag should be cleared after the backup has succeeded. |
PERMIT::DELETE | Owner can delete. If the file system does not support this, deletion is enabled via the WRITE flag. |
PERMIT::EVERYONE_ACCESS | Synonym for EVERYONE_READ | EVERYONE_WRITE | EVERYONE_EXEC | EVERYONE_DELETE |
PERMIT::EVERYONE_DELETE | Synonym for DELETE | GROUP_DELETE | OTHERS_DELETE |
PERMIT::EVERYONE_EXEC | Synonym for EXEC | GROUP_EXEC | OTHERS_EXEC |
PERMIT::EVERYONE_READ | Synonym for READ | GROUP_READ | OTHERS_READ |
PERMIT::EVERYONE_READWRITE | Synonym for EVERYONE_READ | EVERYONE_WRITE |
PERMIT::EVERYONE_WRITE | Synonym for WRITE | GROUP_WRITE | OTHERS_WRITE |
PERMIT::EXEC | User/Owner can execute. |
PERMIT::GROUP | Synonym for GROUP_READ | GROUP_WRITE | GROUP_EXEC | GROUP_DELETE |
PERMIT::GROUPID | Allows executables to run with a set group id. |
PERMIT::GROUP_DELETE | Group members can delete. |
PERMIT::GROUP_EXEC | Group members can execute. |
PERMIT::GROUP_READ | Group members can read. |
PERMIT::GROUP_WRITE | Group members can write. |
PERMIT::HIDDEN | Recommends that the file is hidden from view by default. |
PERMIT::INHERIT | Inherit permissions from parent folder and logical OR them with preset permission flags. |
PERMIT::NETWORK | File is hosted on another machine. |
PERMIT::OFFLINE | File content for this networked file has not been cached on the local PC. |
PERMIT::OTHERS | Synonym for OTHERS_READ | OTHERS_WRITE | OTHERS_EXEC | OTHERS_DELETE |
PERMIT::OTHERS_DELETE | Others can delete. |
PERMIT::OTHERS_EXEC | Others can execute. |
PERMIT::OTHERS_READ | Others can read. |
PERMIT::OTHERS_WRITE | Others can write. |
PERMIT::PASSWORD | File is password protected. |
PERMIT::READ | User/Owner has read access. This will not allow compiled code to be executed. |
PERMIT::USER | Synonym for READ | WRITE | EXEC | DELETE |
PERMIT::USERID | Allows executables to run with a set user id. |
PERMIT::USER_EXEC | Synonym for EXEC |
PERMIT::USER_READ | Synonym for READ |
PERMIT::USER_WRITE | Synonym for WRITE |
PERMIT::WRITE | User/Owner can write. |
Field | Type | Description |
---|---|---|
OriginalSize | LARGE | Original size of the file |
CompressedSize | LARGE | Compressed size of the file |
Next | struct CompressedItem * | Used only if this is a linked-list. |
Path | CSTRING | Path to the file (includes folder prefixes). Archived folders will include the trailing slash. |
Permissions | PERMIT | Original permissions - see PERMIT flags. |
UserID | LONG | Original user ID |
GroupID | LONG | Original group ID |
OthersID | LONG | Original others ID |
Flags | FL | FL flags |
Created | struct DateTime | Date and time of the file's creation. |
Modified | struct DateTime | Date and time last modified. |
Field | Type | Description |
---|---|---|
FeedbackID | FDB | Set to one of the FDB event indicators |
Index | LONG | Index of the current file |
Path | CSTRING | Name of the current file/path in the archive |
Dest | CSTRING | Destination file/path during decompression |
Progress | LARGE | Progress indicator (byte position for the file being de/compressed). |
OriginalSize | LARGE | Original size of the file |
CompressedSize | LARGE | Compressed size of the file |
Year | WORD | Year of the original file's datestamp. |
Month | WORD | Month of the original file's datestamp. |
Day | WORD | Day of the original file's datestamp. |
Hour | WORD | Hour of the original file's datestamp. |
Minute | WORD | Minute of the original file's datestamp. |
Second | WORD | Second of the original file's datestamp. |
Class Info | |
---|---|
ID | ID_COMPRESSION |
Category | Data |
Include | modules/compression.h |
Version | 1 |