Overview

Skill Level: Advanced

Familiarity with C language and IBM Z machine architecture is an advantage.

We explore invoking and tuning the DFLTCC machine instruction in z15 to optimise data compression using the DEFLATE algorithm. We use a wrapper written in LE C to manage and manipulate the options for deflating and inflating data streams.

Ingredients

This recipe requires IBM z15 hardware and z/OS 2.3 or higher with XL C/C++ chargeable feature at level 2.3.1 or higher.

Step-by-step

  1. Introduction

    When IBM released the z15 mainframe processor in 2019, it moved its hardware implementation of the DEFLATE compression algorithm (see: https://tools.ietf.org/html/rfc1951) from the zEDC PCI card to an on-chip machine instruction, DFLTCC.  The documentation for this new instruction in “IBM z/Architecture Principles of Operation” (see: https://www.ibm.com/support/pages/zarchitecture-principles-operation-6) runs to an eye-watering 45 pages, making DFLTCC more of a “silicon subroutine” than a machine instruction.

    As comprehensive as the documentation is, it is not easy to divine how to prepare the parameter block and input/output buffers to produce the outcome one might be seeking.  Accordingly, this recipe attempts to simplify and unify the guidance on how to use this instruction to deflate and inflate data streams.

    It should be noted that z/OS also provides an implementation of the open source “zlib” library in C, and corresponding Java classes.  Examples of the use of these interfaces are given at the end of this recipe.

    This recipe builds a comprehensive C application capable of deflating (i.e. compressing) or inflating (i.e. expanding) any z/OS UNIX file.  This code could be adapted to process MVS files or other data sources.  Specific aspects of this code that deal with manipulating the DFLTCC instruction are called out in the steps below

  2. Invoking DFLTCC

    The DFLTCC instruction takes 3 registers as arguments.  The first two registers must be even registers and they are treated as even-odd register pairs, where the even register (R) is a pointer to a buffer and the odd register (R+1) holds a length value.  The third register is not required to be an even register.  It holds a buffer address.

    In addition, general purpose register 0 (GPR0) is used to select the “function code” and GPR1 points to a parameter block for every function.

    The parameter block is required to be doubleword-aligned (i.e. address is a multiple of 8).

    Function code 0 populates a 32-byte parameter block with a 128-bit mask identifying which function codes are supported and a 16-bit mask identifying which parameter block formats are supported by function codes other than function code 0.  The 3 register arguments must be supplied, but they are not referenced.

    At the time of writing, function codes 0 (of course), 1 (= generate dynamic Huffman table = GDHT), 2 (= deflate = CMPR) and 4 (= inflate = XPND) are supported.  A single parameter block format, “format 0” (PB0), is supported.

    Function code 1 requires a 384-byte parameter block (discussed later).  The second register argument is used to pass the address (in R) and length (in R+1) of a buffer of “training data” used by the GDHT function to create a compressed Huffman table in the supplied parameter block.  The idea is that this parameter block will then be passed to the CMPR function to compress the same (or similar) buffer of data.  The first and third register arguments are required but not referenced by function code 1.

    Function codes 2 and 4 require a 1536-byte parameter block.  The first register argument is used to pass the address and length of a destination buffer, the second register argument holds the address and length of a source buffer, and the third register argument points to a 32kB, 4k-page-aligned, circular history buffer (if selected).

    The DEFLATE standard allows for repeated symbols to be encoded as a (back-distance, length) pair, referring to uncompressed bytes in the “past” (i.e. already compressed or expanded).  For this to work, any implementation of DEFLATE must keep up to 32kB of “history”.  DFLTCC provides two mechanisms to do this.  You must either (a) make sure that sufficient history is still addressable “to the left” of the current buffer being compressed/expanded, or (b) you can provide a separate 32kB buffer, which the DFLTCC instruction uses as a circular history buffer.  In most circumstances, this second option is less complex to manage.  To invoke this option, we provide the address of the 32kB circular buffer in the third register argument and use a function code of 2 + 128 = 130 for the CMPR operation, or 4 + 128 = 132 for the XPND operation.

  3. The Format 0 parameter block (PB0)

    At the time of writing, a single parameter block format is supported by the GDHT, CMPR and XPND functions of the DFTLCC instruction.

    This parameter block format can be defined as a C structure as follows:

    #pragma pack(packed)
    typedef struct {
    uint16_t pbvn; /* parameter block version number */

    uint32_t mvn : 8; /* model version number */
    uint32_t ribm1 : 24; /* reserved for IBM use */

    uint32_t : 15; /* reserved */
    uint32_t cf : 1; /* continuation flag */
    uint32_t : 16; /* reserved */

    uint8_t reserved1[4]; /* reserved */

    uint32_t : 16; /* reserved */
    uint32_t nt : 1; /* new task */
    uint32_t : 1; /* reserved */
    uint32_t cvt : 1; /* check value type */
    uint32_t : 1; /* reserved */
    uint32_t htt : 1; /* Huffman table type */
    uint32_t bcf : 1; /* block continuation flag */
    uint32_t bcc : 1; /* block closing control */
    uint32_t bhf : 1; /* block header final bit */
    uint32_t : 1; /* reserved */
    uint32_t : 1; /* reserved */
    uint32_t dhtgc : 1; /* reserved */
    uint32_t : 5; /* reserved */

    uint32_t : 5; /* reserved */
    uint32_t sbb : 3; /* sub-byte boundary */
    uint32_t oesc : 8; /* operation-ending supplemental code */
    uint32_t : 12; /* reserved */
    uint32_t ifs : 4; /* incomplete function status */

    uint32_t ifl : 16; /* incomplete function length */
    uint32_t : 16; /* reserved */

    uint8_t reserved2[18]; /* reserved */

    uint32_t hl : 16; /* history length */
    uint32_t : 1; /* reserved */
    uint32_t ho : 15; /* history offset */

    uint32_t cv : 32; /* check value */

    uint32_t eobs : 15; /* end-of-block symbol */
    uint32_t : 1; /* reserved */
    uint32_t eobl : 4; /* end-of-block length */
    uint32_t : 12; /* reserved */

    uint32_t : 4; /* reserved */
    uint32_t cdhtl : 12; /* compressed dynamic Huffman table length */
    uint32_t : 16; /* reserved */

    uint8_t reserved3[4]; /* reserved */

    uint8_t cdht[288]; /* compressed dynamic Huffman table */

    uint8_t reserved4[24]; /* reserved */

    uint64_t ribm2; /* reserved for IBM use */

    uint8_t csb[1152]; /* continuation state buffer */
    } DFLTCC_PB0, *PDFLTCC_PB0;

    How to initialise this structure and which bits to flip to invoke certain options will be discussed in subsequent steps.

  4. Raising and Training a Dynamic Huffman Table

    The DEFLATE compression algorithm contemplates two types of dictionary to achieve compression.  The over-arching principle is that frequently-occurring code-points in the uncompressed data will be encoded using fewer bits than rarely-occurring code-points, that may be assigned encodings that are longer than 8 bits, but the net effect (hopefully, but not guaranteed) is a smaller compressed file than the uncompressed file.

    The options for the dictionary that maps uncompressed code-points to their compressed form are (a) a “fixed” Huffman table (FHT), defined by the DEFLATE standard, or (b) a “dynamic” Huffman table (DHT), specific to the input data stream, which is optimal (or close to it) for this specific input.

    The principle advantage of the FHT is that it is standard, hence it does not need to be embedded in the output compressed file.  The disadvantage is that it does not provide much compression, particularly for EBCDIC text, for example.  For small data streams, however, a FHT might result in better compression than a DHT.

    The advantage of a DHT is that it is pruned and tuned to the input it is being asked to compress, but it must be embedded into the output data stream, so it is available to an “expander”.  The other slight nuance is that there is nothing stopping you from using a DHT trained against one data stream to compress a different data stream.  If you are going to do this, you need to create a “universal” DHT, so the dictionary has an entry for all possible code-points.  A universal DHT is slightly larger than a “local” DHT, but can be used to compress arbitrary input.

    Now let’s assume we are going to compress a data stream and we want to use a DHT.  How do we invoke the GDHT function to build a DHT?

    First, define a parameter block as an array of 192 64-bit integers (= 1536 bytes), because we will use the same parameter block for subsequent CMPR operations.  This will ensure the parameter block is doubleword-aligned.  Then set the parameter block to nulls (x’00’).

        uint64_t    dword_block[192];
    uint8_t *parameter_block;
    PDFLTCC_PB0 ppb0;

    parameter_block = (uint8_t *)(void *)dword_block;
    memset(parameter_block, 0, sizeof(dword_block));
    ppb0 = (PDFLTCC_PB0)(void *)dword_block;

    Now select Huffman table type of “dynamic”, by setting the HTT bit in the parameter block to 1.  If we intend to use the generated DHT to compress more data than we are using for “training” (or we are not sure), select a universal DHT by setting the “dynamic Huffman table generation control” (DHTGC) bit in the parameter block to 0, otherwise set it to 1.  Finally, load the training data (i.e. up to 32kB of data from the uncompressed data stream) into a buffer, set the second even-odd register pair to the address and length of this buffer, and invoke the DFLTCC instruction with a function code of 1.

        int cc, func;

    ppb0->htt = 1;
    ppb0->dhtgc = 1;
    if (trainlen < buflen) {
    ppb0->dhtgc = 0; /* generate a universal Huffman table */
    }
    func = 1;
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    NULL);

    Our generic “wrapper” function to invoke the DFLTCC instruction looks like this:

    int DFLT(int        function_code,
    uint8_t *parameter_block,
    uint8_t **poperand_1,
    uint32_t *poperand_1_length,
    uint8_t **poperand_2,
    uint32_t *poperand_2_length,
    uint8_t *operand_3)
    {
    uint32_t r0, cc, operand_1_length, operand_2_length;
    uint8_t *operand_1, *operand_2;

    r0 = (uint32_t)function_code;
    cc = 0;
    operand_1 = *poperand_1;
    operand_1_length = *poperand_1_length;
    operand_2 = *poperand_2;
    operand_2_length = *poperand_2_length;

    __asm( " DFLTCC 6,8,10 \\n"
    " IPM 11 \\n"
    " SRL 11,28 "
    : "=NR:r11"(cc),
    "+NR:r6"(operand_1),
    "+NR:r7"(operand_1_length),
    "+NR:r8"(operand_2),
    "+NR:r9"(operand_2_length)
    : "NR:r0"(r0),
    "NR:r1"(parameter_block),
    "NR:r10"(operand_3)
    : );

    *poperand_1 = operand_1;
    *poperand_1_length = operand_1_length;
    *poperand_2 = operand_2;
    *poperand_2_length = operand_2_length;

    return (int)cc;
    }

    DFLTCC sets a condition code of 0 if the DHT was computed and loaded into the parameter block successfully, otherwise it sets a condition code of 2.

  5. Compressing a Data Stream

    To keep things simple, assume we are going to read an input file into a fixed-length buffer, noting that the final read may not fill that buffer.  We will use up to 32kB of the buffer to train a DHT (as above), unless the number of uncompressed bytes available is less than 512, in which case we will specify that the standard “fixed” Huffman table is to be used.  Each buffer of input data will be used to produce a single compressed block, for which we will provide an output buffer that is 2kB bigger than the input buffer (just to be safe).

    We will use the parameter block returned by the GDHT call as a starting point, unless the only input buffer (i.e. small input file) has less than 512 bytes of data, in which case we will start with a parameter block initialised to nulls (x’00’).

    When compressing the first block of input data, the parameter block is prepared as follows:

    • We set the “new task” (NT) bit in the parameter block to 1.  This causes the DFLTCC instruction to emit a 3-bit DEFLATE header into the output buffer and it causes the “check value” (CV) field in the parameter block to be properly initialised.
    • We set the “check value type” (CVT) bit in the parameter block to indicate the type of check value we want computed – 0 for CRC-32 or 1 for Adler.
    • We set the “block closing control” (BCC) bit to 1 to cause an “end of block” (EOB) symbol to be written at the end of the compressed output

    Now we invoke DFLTCC in a loop, passing the output buffer in the first register argument (with length set to the maximum space available), the input buffer in the second register argument (with length set to the number of uncompressed bytes available), and a pointer to a 32kB history buffer in the third register argument, until we receive a condition code of 0, or an error occurs.

        ppb0->nt = 1;        /* start a new task                    */
    ppb0->cvt = 1; /* compute Adler check value */
    ppb0->bcc = 1; /* emit EOB at end of compressed block */
    op1 = outBuffer;
    op1len = outlen; /* maximum length of output buffer */
    op2 = inBuffer;
    op2len = buflen; /* number of bytes read from input */
    cc = 3;
    while (cc == 3) {
    func = 2 + 128; /* use a circular history buffer */
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    historyBuffer);
    }
    if (cc == 0) {
    /* success */
    }
    else {
    /* something went wrong */
    }

    The 4k page-aligned history buffer is allocated as follows:

        int     rc;
    uint8_t *historyBuffer;

    __asm(" SYSSTATE ARCHLVL=2 \\n"
    " STORAGE OBTAIN,LENGTH=32768,BNDRY=PAGE "
    : "+NR:r1"(historyBuffer),
    "+NR:r15"(rc)
    :
    : "r0", "r14");

    In our loop invoking DFLTCC we should expect that a condition code of 3 might be returned, in which case we simply re-drive DFLTCC without modifying any parameters.  If we receive a condition code of 1, the output buffer is not big enough to hold the compressed block.  This should never happen, given our output buffer is larger than the input, so we treat this as an error.  Any other error condition will be indicated by a condition code of 2.

    When we receive a condition code of 0 for the first block of input, we should find that the length of the second register argument has been set to zero (i.e. the input is exhausted) and the length of the first register argument has been decremented by the number of “complete” bytes written to the output buffer, noting that the compressed symbols will not generally align to byte boundaries.  So we write the number of complete output bytes (outlen – op1len) to our output file, and if the “sub-byte boundary” (SBB) field in the parameter block is greater than zero (meaning the compressed block did not end on a byte boundary), we move the byte immediately after the complete output bytes to the beginning of our output buffer, ready for it to be completed by the generation of the next compressed block.

    To process the next chunk of uncompressed input, we read it into the input buffer, set the buffer length accordingly, set the output buffer length to the maximum value again.  If we have sufficient input data to generate a new DHT, we invoke the GDHT function (as in the previous step) to do so.  Apart from HTT, we do not modify any other elements of the parameter block, unless this is the last block of input (i.e. the input file is exhausted), in which case, we set the “block header final” (BHF) bit in the parameter block to 1.  This causes the DFLTCC instruction to emit a DEFLATE header with the BFINAL bit set to 1, indicating to an “expander” that this compressed block is the final block to be processed.

    Once we have used DFLTCC to produce the last compressed block and we have written out the completed bytes of that compressed block, we check the SBB field of the parameter block.  If SBB > 0 we write the byte immediately after the last complete byte of the output buffer to our output file, because it contains some bits of the final symbol of the final compressed block.

  6. Inflating a Data Stream

    Suppose we have an input file of compressed blocks.  How do we reconstitute the original uncompressed file?  Mercifully, this process is simpler than the compression process, because the XPND function automatically handles the different types and lengths of compressed blocks (e.g. some FHT, some DHT, some not compressed, potentially).  All we have to do is feed in the compressed data and write out the uncompressed data to a file.

    Using the same buffers and 1536-byte parameter block (initialised to nulls) as for the CMPR function, we prepare the parameter block as follows:

    •  We set the new task (NT) bit to 1.
    •  We set the check value type (CVT) to 0 (for CRC-32) or 1 (for Adler) check value calculation.

    Now we read in the first input buffer full of compressed data and invoke DFLTCC in a loop, until we receive a condition code of 0.  Along the way, as you can imagine, other conditions will arise:

    If we receive a condition code of 3, we re-drive the DFLTCC instruction without changing the parameter block.

    If we receive a condition code of 1, our output buffer is full.  In this case, we should see that op2len has been decremented to zero, so we write out the entire buffer, set op2 to point to the beginning of the output buffer and set op2len to the size of the output buffer.

    If we receive a condition code of 2 and OESC = x’00’, the input buffer is exhausted.  Before we read in more compressed data from the input file, we should check the SBB field in the parameter block.  If SBB > 0 (and in such a case, op1len must be greater than zero, i.e. not all complete bytes of input have been processed), we should move the unprocessed bytes (if any) to the beginning of our input buffer and read in more compressed data immediately after those bytes.  At the time of writing, I have not observed a completion code of 2, OESC of x’00’ and SBB > 0, but the description of the XPND function does not prohibit this situation, so we should allow for it.

    If we receive a condition code of 2 and OESC is not equal to x’00’, an error has occurred and the OESC value indicates the cause of the problem (see the description of the DFLTCC instruction in z/Architecture Principles of Operation for details).

    A somewhat simplified rendition of this processing loop follows:

        parameter_block = (uint8_t *)(void *)dword_block;
    memset(parameter_block, 0, sizeof(dword_block));
    ppb0 = (PDFLTCC_PB0)(void *)dword_block;
    ppb0->nt = 1;
    ppb0->cvt = checkValue;

    done = FALSE;
    /* read in first chunk of input file */
    buflen = (int)fread(inBuffer, 1, maxbuflen, fin);
    bytesRead += buflen;
    op1 = outBuffer;
    op1len = outlen;
    op2 = inBuffer;
    op2len = buflen;
    while (!done) {
    func = 4 + 128; /* use a circular history buffer */
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    historyBuffer);
    if (cc == 0) { /* normal completion */
    writeOut(outBuffer, outlen - op1len, fout, &bytesWritten);
    done = TRUE;
    }
    else if (cc == 3) {
    /* leave evrything as-is and go around again */
    done = FALSE;
    }
    else if (cc == 2 && ppb0->oesc == 0x00) {
    /* input buffer needs filling, but may not have been emptied */
    memmove(inBuffer, inBuffer + buflen - op2len, op2len);
    newlen = fread(inBuffer + op2len, 1, buflen - op2len, fin);
    buflen = op2len + newlen;
    op2 = inBuffer;
    op2len = buflen;
    }
    else if (cc == 1) {
    /* output buffer needs emptying */
    writeOut(outBuffer, outlen - op1len, fout, &bytesWritten);
    /* move remainder of output buffer to the start */
    if (op1len > 0) {
    printf("inflate: *** shouldn't happen - output buffer not full\\n");
    memmove(outBuffer, outBuffer+outlen-op1len, op1len);
    }
    op1 = outBuffer + op1len;
    op1len = outlen - op1len;
    }
    else {
    fprintf(stderr, "ERROR occurred during inflation, cc=%u, OESC=x'%02X'\\n",
    cc, ppb0->oesc);
    done = TRUE;
    }
    }
  7. Tying it all together - the complete DEFLATE/inflate program

    Here is a complete C program, z15dflt.c, with various options to customise its behaviour.  This program will compress (deflate) an input file, or expand (inflate) it.  This program also implements an in-line history buffer (enabled by the -h option) and it will deflate by writing a single compressed block (enabled by the -o option), rather than writing a compressed block for each buffer-full of input data it reads from the uncompressed input file.  Once compiled, invoke the program with no parameters to see a “usage” display, which documents all the available options.

    /* z15dflt.c - Call z15 DFLTCC machine instruction */

    #define _POSIX_SOURCE
    #define _XOPEN_SOURCE
    #include <ceeedcct.h>
    #include <leawi.h>
    #include <signal.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <sys/stat.h>

    #ifndef FALSE
    #define FALSE 0
    #endif
    #ifndef TRUE
    #define TRUE -1
    #endif

    #pragma pack(packed)
    typedef struct {
    uint16_t pbvn; /* parameter block version number */

    uint32_t mvn : 8; /* model version number */
    uint32_t ribm1 : 24; /* reserved for IBM use */

    uint32_t : 15; /* reserved */
    uint32_t cf : 1; /* continuation flag */
    uint32_t : 16; /* reserved */

    uint8_t reserved1[4]; /* reserved */

    uint32_t : 16; /* reserved */
    uint32_t nt : 1; /* new task */
    uint32_t : 1; /* reserved */
    uint32_t cvt : 1; /* check value type */
    uint32_t : 1; /* reserved */
    uint32_t htt : 1; /* Huffman table type */
    uint32_t bcf : 1; /* block continuation flag */
    uint32_t bcc : 1; /* block closing control */
    uint32_t bhf : 1; /* block header final bit */
    uint32_t : 1; /* reserved */
    uint32_t : 1; /* reserved */
    uint32_t dhtgc : 1; /* reserved */
    uint32_t : 5; /* reserved */

    uint32_t : 5; /* reserved */
    uint32_t sbb : 3; /* sub-byte boundary */
    uint32_t oesc : 8; /* operation-ending supplemental code */
    uint32_t : 12; /* reserved */
    uint32_t ifs : 4; /* incomplete function status */

    uint32_t ifl : 16; /* incomplete function length */
    uint32_t : 16; /* reserved */

    uint8_t reserved2[18]; /* reserved */

    uint32_t hl : 16; /* history length */
    uint32_t : 1; /* reserved */
    uint32_t ho : 15; /* history offset */

    uint32_t cv : 32; /* check value */

    uint32_t eobs : 15; /* end-of-block symbol */
    uint32_t : 1; /* reserved */
    uint32_t eobl : 4; /* end-of-block length */
    uint32_t : 12; /* reserved */

    uint32_t : 4; /* reserved */
    uint32_t cdhtl : 12; /* compressed dynamic Huffman table length */
    uint32_t : 16; /* reserved */

    uint8_t reserved3[4]; /* reserved */

    uint8_t cdht[288]; /* compressed dynamic Huffman table */

    uint8_t reserved4[24]; /* reserved */

    uint64_t ribm2; /* reserved for IBM use */

    uint8_t csb[1152]; /* continuation state buffer */
    } DFLTCC_PB0, *PDFLTCC_PB0;

    /* processing option constants */
    #define Z15ACT_DEFLATE 0
    #define Z15ACT_INFLATE 1
    #define Z15BUF_DEFAULT 32
    #define Z15BUF_MAX 1000000
    #define Z15BUF_TRAINMAX 32
    #define Z15BUF_MARGIN 2 /* allow extra 2kB of output buffer */
    #define Z15CV_CRC 0
    #define Z15CV_ADLER 1
    #define Z15FMT_PB0 0
    #define Z15HBT_INLINE 0
    #define Z15HBT_CIRCULAR 128
    #define Z15HTT_FIXED 0
    #define Z15HTT_DYNAMIC 1
    #define Z15HTT_SHORTBUF 512 /* use FHT for short input buffer */
    #define Z15OP_QAF 0
    #define Z15OP_GDHT 1
    #define Z15OP_CMPR 2
    #define Z15OP_XPND 4
    #define Z15VER_QUIET 0
    #define Z15VER_VERBOSE 1

    /* global variables */
    int verbose;
    uint8_t *inBuffer, *outBuffer, *historyBuffer;
    FILE *fin, *fout;
    int nDfltccCalls, inMax, outMax;
    double timeInDfltcc;

    /* private function prototypes */
    void abendHandler(int sig);
    int availableFunctions(uint8_t *status);
    char *bitString(uint32_t val, uint32_t sigBits, uint32_t len);
    void cleanUp(void);
    uint32_t deflate(int checkValue,
    int tableType,
    int historyBufferType,
    int oneBigBlock,
    int bufferLength,
    int trainLength,
    off_t infileSize);
    void dump(uint8_t *buffer, uint32_t bufferLength);
    void dumpPB0(DFLTCC_PB0 *ppb, uint32_t pbLength);
    int getBuffers(int bufferLength, int historyBufferType);
    uint32_t inflate(int checkValue,
    int historyBufferType,
    int bufferLength);
    int isAvailable(uint8_t *status, int function_code);
    int isFormat(uint8_t *status, int format);
    int openFiles(char *infile, char *outfile, off_t *pinfileSize);
    double timeDifference(uint64_t t0, uint64_t t1);
    int writeOut(uint8_t *buffer, int length, FILE *f, uint64_t *pwritten);
    int DFLT(int function_code,
    uint8_t *parameter_block,
    uint8_t **poperand_1,
    uint32_t *poperand_1_length,
    uint8_t **poperand_2,
    uint32_t *poperand_2_length,
    uint8_t *operand_3);

    int main(int argc, char **argv)
    {
    int rc, ok;
    int action, checkValue, hbt, obb, tableType, bufferLength, trainLength;
    char *inFile, *outFile;
    uint64_t t0, t1;
    double elapsed;
    off_t infileSize;

    /* initialise variables before trapping exceptions */
    inBuffer = NULL;
    inMax = 0;
    outBuffer = NULL;
    outMax = 0;
    historyBuffer = NULL;
    fin = NULL;
    fout = NULL;
    nDfltccCalls = 0;
    timeInDfltcc = 0.0;
    signal(SIGABND, abendHandler);
    signal(SIGABRT, abendHandler);
    signal(SIGFPE, abendHandler);
    signal(SIGILL, abendHandler);
    signal(SIGSEGV, abendHandler);
    signal(SIGTERM, abendHandler);
    signal(SIGIOERR, abendHandler);

    /* process command-line arguments */
    optind = opterr = 0;
    action = Z15ACT_DEFLATE; /* deflate */
    checkValue = Z15CV_CRC; /* CRC */
    bufferLength = Z15BUF_DEFAULT;
    trainLength = Z15BUF_DEFAULT;
    hbt = Z15HBT_CIRCULAR; /* circular history buffer */
    tableType = Z15HTT_DYNAMIC; /* dynamic Huffman table */
    verbose = Z15VER_QUIET; /* quiet */
    obb = FALSE; /* 1 compressed block per uncompressed input buffer */
    ok = TRUE;
    while ((rc = getopt(argc, argv, "ab:cdfhiot:vy")) != -1) {
    switch (rc) {
    case 'a': /* Adler check value */
    checkValue = Z15CV_ADLER;
    break;
    case 'b': /* input buffer size */
    bufferLength = atoi(optarg);
    if (bufferLength < 1 || bufferLength > Z15BUF_MAX) {
    fprintf(stderr, "ERROR: Invalid input buffer length, %s\\n", optarg);
    ok = FALSE;
    }
    break;
    case 'c': /* CRC check value */
    checkValue = Z15CV_CRC;
    break;
    case 'd': /* deflate */
    action = Z15ACT_DEFLATE;
    break;
    case 'f': /* fixed Huffman table */
    tableType = Z15HTT_FIXED;
    break;
    case 'h': /* use in-line history buffer */
    hbt = Z15HBT_INLINE;
    break;
    case 'i': /* inflate */
    action = Z15ACT_INFLATE;
    break;
    case 'o': /* one big block of compressed output */
    obb = TRUE;
    break;
    case 't': /* training buffer size */
    trainLength = atoi(optarg);
    if (trainLength < 1 || trainLength > Z15BUF_TRAINMAX) {
    fprintf(stderr, "ERROR: Invalid training buffer length, %s\\n", optarg);
    ok = FALSE;
    }
    break;
    case 'v': /* verbose output */
    verbose = Z15VER_VERBOSE;
    break;
    case 'y': /* dynamic Huffman table */
    tableType = Z15HTT_DYNAMIC;
    break;
    case '?': /* invalid option */
    fprintf(stderr, "ERROR: Invalid option, -%c\\n", optopt);
    ok = FALSE;
    break;
    default:
    break;
    }
    }
    if (optind != argc - 2) {
    fprintf(stderr, "ERROR: %u parameters supplied. 2 are required\\n", argc - optind);
    ok = FALSE;
    }
    else {
    inFile = argv[optind++];
    outFile = argv[optind++];
    }
    if (!ok) {
    printf(" \\n");
    printf("Usage: [-acdfiovy -b n -t n] infile outfile\\n");
    printf(" \\n");
    printf(" Options:\\n");
    printf(" -a compute Adler check value\\n");
    printf(" -b n set input buffer size to n kB (default=32)\\n");
    printf(" -c compute CRC check value (default)\\n");
    printf(" -d deflate input file into output file (default)\\n");
    printf(" -f use fixed Huffman table\\n");
    printf(" -h use in-line history buffer\\n");
    printf(" -i inflate input file into output file\\n");
    printf(" -o emit compressed file as a single block\\n");
    printf(" -t n set training buffer size to n kB (default=32)\\n");
    printf(" -v verbose output\\n");
    printf(" -y use dynamic Huffman table (default)\\n");
    printf(" \\n");
    return 8;
    }

    /* if using in-line history, increase buffer size by 32kB */
    if (hbt == Z15HBT_INLINE) {
    bufferLength += 32;
    }

    /* no point training with a longer buffer than the input buffer */
    if (action == Z15ACT_DEFLATE && trainLength > bufferLength) {
    fprintf(stderr, "WARNING: Training truncated to input buffer length, %ukB\\n",
    bufferLength);
    trainLength = bufferLength;
    }

    if (getBuffers(bufferLength, hbt)) {
    if (openFiles(inFile, outFile, &infileSize)) {
    /* print out selected processing options */
    printf(". Operation: %s\\n", (action == Z15ACT_DEFLATE) ? "DEFLATE" : "inflate");
    printf(". Input file: %s\\n", inFile);
    printf(". Output file: %s\\n", outFile);
    printf(". Buffer size: %u kB\\n", bufferLength);
    printf(". Check value: %s\\n", (checkValue == Z15CV_ADLER) ? "Adler" : "CRC-32");
    printf(". History buffer: %s\\n", (hbt == Z15HBT_INLINE) ? "In-line" : "Circular");
    if (action == Z15ACT_DEFLATE) {
    printf(". Training size: %u kB\\n", trainLength);
    printf(". Huffman table: %s\\n", (tableType == Z15HTT_FIXED) ? "Fixed" : "Dynamic");
    printf(". Compress to: %s\\n", obb ? "One compressed block" : "Many blocks (one per input buffer)");
    }

    __asm(" STCKF %0" : "=m"(t0) : : );
    if (action == Z15ACT_DEFLATE) {
    deflate(checkValue, tableType, hbt, obb, bufferLength, trainLength, infileSize);
    }
    else {
    inflate(checkValue, hbt, bufferLength);
    }
    __asm(" STCKF %0" : "=m"(t1) : : );
    printf("Calls to DFLTCC: %u\\n", nDfltccCalls);
    printf("Time in DFLTCC: %.6f s\\n", timeInDfltcc);
    elapsed = timeDifference(t0, t1);
    printf("Total elapsed time: %.6f s\\n", elapsed);
    }
    }

    cleanUp();

    return 0;
    }

    void abendHandler(int sig)
    {
    _CEECIB *cib_ptr;
    _FEEDBACK cibfc;

    printf("*** Exception occurred, signal=%d\\n", sig);

    /* get abend code */
    CEE3CIB(NULL, &cib_ptr, &cibfc);

    /* verify that CEE3CIB was successful */
    if ( _FBCHECK ( cibfc , CEE000 ) != 0 ) {
    printf("CEE3CIB failed with message number %d\\n",
    cibfc.tok_msgno);
    }
    else {
    printf("System completion code: %03X\\n",
    (((unsigned int)(cib_ptr->cib_abcd)) >> 12) & 0xfff);
    }
    signal(sig, SIG_DFL);

    /* clean up */
    cleanUp();

    /* get out */
    exit(8);
    }

    int availableFunctions(uint8_t *status)
    {
    uint64_t dword_block[4];
    uint8_t *parameter_block, *op1, *op2;
    uint32_t op1len, op2len, rc;

    parameter_block = (uint8_t *)(void *)dword_block;
    memset(parameter_block, 0, sizeof(dword_block));

    op1 = NULL;
    op1len = 0;
    op2 = NULL;
    op2len = 0;
    if (verbose == Z15VER_VERBOSE) {
    printf("availableFunctions: DFLTCC-QAF\\n");
    }
    rc = DFLT(Z15OP_QAF,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    NULL);
    if (rc == 0) {
    memcpy(status, parameter_block, sizeof(dword_block));
    }

    return (rc == 0);
    }

    char *bitString(uint32_t val, uint32_t sigBits, uint32_t len)
    {
    static char text[40];
    uint32_t i, j, mask;

    mask = 1 << (sigBits - 1);
    j = 0;
    for (i = 0; i < len; i++) {
    if (mask & val) {
    text[j++] = '1';
    }
    else {
    text[j++] = '0';
    }
    if (i % 8 == 7) {
    text[j++] = ' ';
    }
    mask >>= 1;
    }
    text[j++] = '\\\\00';

    return text;
    }

    void cleanUp(void)
    {
    if (inBuffer != NULL) {
    free(inBuffer);
    inBuffer = NULL;
    }

    if (outBuffer != NULL) {
    free(outBuffer);
    outBuffer = NULL;
    }

    if (historyBuffer != NULL) {
    __asm (" SYSSTATE ARCHLVL=2 \\n"
    " STORAGE RELEASE,ADDR=(1),LENGTH=32768 "
    : "+NR:r1"(historyBuffer)
    :
    : "r0", "r14", "r15");
    historyBuffer = NULL;
    }

    if (fin != NULL) {
    fclose(fin);
    fin = NULL;
    }

    if (fout != NULL) {
    fclose(fout);
    fout = NULL;
    }
    }

    uint32_t deflate(int checkValue,
    int tableType,
    int historyBufferType,
    int oneBigBlock,
    int bufferLength,
    int trainLength,
    off_t infileSize)
    {
    uint64_t dword_block[192], bytesRead, bytesWritten;
    uint8_t *parameter_block, *op1, *op2, status[32];
    PDFLTCC_PB0 ppb0;
    uint32_t buflen, outlen, trainlen, op1len, op2len, offset, oldOffset;
    int cc, done, func, ok;

    /* check that the DFLTCC functions we need are available */
    ok = FALSE;
    if (availableFunctions(status)) {
    if (isAvailable(status, Z15OP_CMPR)) {
    ok = TRUE;
    if (tableType == Z15HTT_DYNAMIC) {
    ok = FALSE;
    if (isAvailable(status, Z15OP_GDHT)) {
    ok = TRUE;
    }
    else {
    fprintf(stderr, "ERROR: DFLTCC-GDHT function is not available\\n");
    }
    }
    }
    else {
    fprintf(stderr, "ERROR: DFLTCC-CMPR function is not available\\n");
    }
    ok = FALSE;
    if (isFormat(status, Z15FMT_PB0)) {
    ok = TRUE;
    }
    else {
    fprintf(stderr, "ERROR: Parameter block format 0 is not available\\n");
    }
    }
    else {
    fprintf(stderr, "ERROR: Unable to retrieve available DFLTCC functions\\n");
    }
    if (!ok) {
    return 0;
    }

    parameter_block = (uint8_t *)(void *)dword_block;
    memset(parameter_block, 0, sizeof(dword_block));
    ppb0 = (PDFLTCC_PB0)(void *)dword_block;
    if (checkValue) ppb0->cvt = 1; else ppb0->cvt = 0;

    /*
    * outer loop - read block of uncompressed input and deflate them
    */
    done = FALSE;
    bytesRead = 0L;
    bytesWritten = 0L;
    outlen = (bufferLength + Z15BUF_MARGIN) * 1024;
    ppb0->nt = 1; /* new task */
    buflen = 0;
    offset = 0;
    if (infileSize == 0) done = TRUE;
    while (!done) {
    /* if using in-line history buffer, preserve HL worth of input from previous call */
    oldOffset = offset;
    offset = (historyBufferType == Z15HBT_INLINE) ? ppb0->hl : 0;
    if (offset > 0) {
    memmove(inBuffer, inBuffer + buflen + oldOffset - offset, offset);
    }
    /* read in chunks of input file */
    buflen = (int)fread(inBuffer + offset, 1, bufferLength * 1024 - offset, fin);
    bytesRead += buflen;

    /* sort out whether we are starting a new compressed block */
    if (oneBigBlock) {
    ppb0->bhf = 1; /* only one compressed block, so set BFINAL bit */
    ppb0->bcc = 0; /* don't emit EOB symbol unless this is the last buffer (see below) */
    if (ppb0->nt) ppb0->bcf = 0; else ppb0->bcf = 1; /* emit block header (and DHT) for first buffer */
    }
    else {
    ppb0->bhf = 0; /* don't set BFINAL unless this is the last buffer (see below) */
    ppb0->bcc = 1; /* emit EOB symbol after each buffer is compressed */
    ppb0->bcf = 0; /* emit block header (and DHT) for each buffer */
    }
    if (feof(fin) || bytesRead == infileSize) {
    ppb0->bhf = 1; /* always set BFINAL on last block */
    ppb0->bcc = 1; /* always emit EOB symbol for last buffer */
    }

    if (!ppb0->bcf) { /* we are writing a new compressed block, so we need to make a HTT decision */
    /* if using dynamic Huffman table, and input buffer is long enough,
    * generate a dynamic Huffman table */
    if (tableType == Z15HTT_DYNAMIC && buflen > Z15HTT_SHORTBUF) {
    ppb0->htt = Z15HTT_DYNAMIC;
    trainlen = trainLength * 1024;
    op1 = NULL;
    op1len = 0;
    op2 = inBuffer + offset;
    op2len = buflen < trainlen ? buflen : trainlen;
    /* set Huffman table type to dynamic. This is ignored, but
    * preserved by DFLTCC-GDHT operation, ready for subsequent
    * DFLTCC-CMPR operation */
    ppb0->htt = Z15HTT_DYNAMIC;
    /* if training length is shorter than buffer length or output
    * is one big compressed block, generate a universal dynamic
    * Huffman table */
    ppb0->dhtgc = 1;
    if (trainlen < buflen || oneBigBlock) {
    ppb0->dhtgc = 0; /* generate a universal Huffman table */
    }
    func = Z15OP_GDHT;
    if (verbose == Z15VER_VERBOSE) {
    printf("deflate: DFLTCC-GDHT\\n");
    }
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    NULL);
    if (cc != 0) {
    fprintf(stderr, "ERROR: DFLTCC-GDHT failed, cc=%u, OESC=x'%02X'\\n",
    cc, ppb0->oesc);
    done = TRUE;
    }
    }
    else {
    ppb0->htt = Z15HTT_FIXED;
    }
    }
    if (!done) {
    op1 = outBuffer;
    op1len = outlen;
    op2 = inBuffer + offset;
    op2len = buflen;
    /*
    * inner loop, until CC != 3
    */
    cc = 3;
    while (cc == 3) {
    func = Z15OP_CMPR + historyBufferType;
    if (verbose == Z15VER_VERBOSE) {
    printf("deflate: DFLTCC-CMPR\\n");
    }
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    historyBuffer);
    if (verbose == Z15VER_VERBOSE) {
    /* compressed data */
    printf("Compressed output (len=%u):\\n", outlen - op1len);
    printf(" first 4 bytes: %s\\n",
    bitString(*(uint32_t *)(void *)outBuffer, 32, 32));
    if (outlen - op1len <= 256) {
    dump(outBuffer, outlen - op1len);
    }
    else {
    dump(outBuffer, 128);
    printf("...\\n");
    dump(outBuffer + outlen - op1len - 126, 128);
    }
    printf(" last 2+2 SBB=%u: %s\\n",
    ppb0->sbb,
    bitString(*(uint32_t *)(void *)(outBuffer+outlen-op1len-2), 32, 32));
    }
    }
    if (cc != 0 || feof(fin) || bytesRead == infileSize) {
    done = TRUE;
    }
    if (cc == 0) {
    /* write out compressed bytes */
    if (!writeOut(outBuffer, outlen - op1len, fout, &bytesWritten)) {
    done = TRUE;
    }
    if ((feof(fin) || bytesRead == infileSize) && ppb0->sbb > 0) {
    /* need to write 1 extra byte */
    if (verbose == Z15VER_VERBOSE) {
    printf("Writing out last incomplete byte of compressed data\\n");
    }
    if (!writeOut(outBuffer + outlen - op1len, 1, fout, &bytesWritten)) {
    done = TRUE;
    }
    }
    if (ppb0->sbb > 0) {
    /* move the byte after those processed to the start of the output buffer */
    outBuffer[0] = outBuffer[outlen - op1len];
    }
    }
    }
    }

    printf("Bytes read from input: %llu\\n", bytesRead);
    printf("Bytes written to output: %llu\\n", bytesWritten);
    if (bytesWritten > 0L) {
    printf("Compression ratio: %.1f : 1\\n", ((double)bytesRead) / ((double)bytesWritten));
    }
    if (checkValue == Z15CV_ADLER) {
    printf("Check value (Adler): x'%08X'\\n", ppb0->cv);
    }
    else {
    printf("Check value (CRC-32): x'%08X'\\n", ppb0->cv);
    }

    return ppb0->cv;
    }

    /* define character for non-displayable code points */
    #define NDC '.'

    /* ASCII character set */
    unsigned char asc[] = {
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    ' ', '!', '"', '#', '$', '%', '&', '\\'',
    '(', ')', '*', '+', ',', '-', '.', '/',
    '0', '1', '2', '3', '4', '5', '6', '7',
    '8', '9', ':', ';', '<', '=', '>', '?',
    '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G',
    'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
    'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W',
    'X', 'Y', 'Z', '[', '\\\\',']', '^', '_',
    NDC, 'a', 'b', 'c', 'd', 'e', 'f', 'g',
    'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o',
    'p', 'q', 'r', 's', 't', 'u', 'v', 'w',
    'x', 'y', 'z', '{', '|', '}', '~', NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC};

    /* EBCDIC character set 500 */
    unsigned char ebc[] = {
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    ' ', NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, '[', '.', '<', '(', '+', '!',
    '&', NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, ']', '$', '*', ')', ';', '^',
    '-', '/', NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, '|', ',', '%', '_', '>', '?',
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, '`', ':', '#', '@', '\\'','=', '"',
    NDC, 'a', 'b', 'c', 'd', 'e', 'f', 'g',
    'h', 'i', NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, 'j', 'k', 'l', 'm', 'n', 'o', 'p',
    'q', 'r', NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, '~', 's', 't', 'u', 'v', 'w', 'x',
    'y', 'z', NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, NDC, NDC, NDC, NDC, NDC,
    NDC, NDC, NDC, '|', NDC, NDC, NDC, NDC,
    '{', 'A', 'B', 'C', 'D', 'E', 'F', 'G',
    'H', 'I', NDC, NDC, NDC, NDC, NDC, NDC,
    '}', 'J', 'K', 'L', 'M', 'N', 'O', 'P',
    'Q', 'R', NDC, NDC, NDC, NDC, NDC, NDC,
    '\\\\',NDC, 'S', 'T', 'U', 'V', 'W', 'X',
    'Y', 'Z', NDC, NDC, NDC, NDC, NDC, NDC,
    '0', '1', '2', '3', '4', '5', '6', '7',
    '8', '9', NDC, NDC, NDC, NDC, NDC, NDC};

    void dump(uint8_t *buffer, uint32_t bufferLength)
    {
    unsigned long i, j;
    unsigned char ascii[17], ebcdic[17];

    for (i = 0; i < bufferLength; i++) {
    if (i % 16 == 0) {
    memset(ascii, ' ', 16);
    ascii[16] = '\\\\00';
    memset(ebcdic, ' ', 16);
    ebcdic[16] = '\\\\00';
    printf("%06X: ", i);
    }
    printf("%02X", buffer[i]);
    if (i % 4 == 3 && i % 16 < 15) {
    printf(" ");
    }
    ascii[i % 16] = asc[(int)buffer[i]];
    ebcdic[i % 16] = ebc[(int)buffer[i]];
    if (i % 16 == 15 || i == bufferLength - 1) {
    for (j = (i % 16) + 1; j < 16; j++) {
    printf(" ");
    if (j % 4 == 3 && j < 15) {
    printf(" ");
    }
    }
    // printf(" |%s| |%s|\\n", ascii, ebcdic);
    printf(" |%s|\\n", ebcdic);
    }
    }
    }

    void dumpPB0(DFLTCC_PB0 *ppb, uint32_t pbLength)
    {
    printf("Format 0 parameter block:\\n");
    printf(" PBVN: %-5u MVN: %-5u CF: %-5u NT: %-5u CVT: %-5u HTT: %-5u\\n",
    ppb->pbvn, ppb->mvn, ppb->cf, ppb->nt, ppb->cvt, ppb->htt);
    printf(" BCF: %-5u BCC: %-5u BHF: %-5u DHTGC: %-5u SBB: %-5u OESC: x'%02X'\\n",
    ppb->bcf, ppb->bcc, ppb->bhf, ppb->dhtgc, ppb->sbb, ppb->oesc);
    printf(" IFS: %-5u IFL: %-5u HL: %-5u HO: %-5u CV: x'%08X'\\n",
    ppb->ifs, ppb->ifl, ppb->hl, ppb->ho, ppb->cv);
    printf(" EOBS: b'%-16s' EOBL: %-5u CDHTL: %-5u CDHT:\\n",
    bitString(ppb->eobs, 15, ppb->eobl), ppb->eobl, ppb->cdhtl);
    dump(ppb->cdht, (ppb->cdhtl + 7) / 8);
    if (pbLength > 384) {
    printf(" Continutation state buffer:\\n");
    dump(ppb->csb, 1152);
    }
    }

    int getBuffers(int bufferLength, int historyBufferType)
    {
    int rc = 0;

    inMax = bufferLength * 1024;
    inBuffer = malloc(inMax);
    if (inBuffer == NULL) {
    perror("ERROR: Unable to allocate input buffer");
    return FALSE;
    }

    outMax = (bufferLength + Z15BUF_MARGIN) * 1024;
    outBuffer = malloc(outMax);
    if (outBuffer == NULL) {
    perror("ERROR: Unable to allocate output buffer");
    return FALSE;
    }

    historyBuffer = NULL;
    if (historyBufferType == Z15HBT_CIRCULAR) {
    /* get 32kB of 4K-aligned storage for circular history buffer */
    __asm(" SYSSTATE ARCHLVL=2 \\n"
    " STORAGE OBTAIN,LENGTH=32768,BNDRY=PAGE "
    : "+NR:r1"(historyBuffer),
    "+NR:r15"(rc)
    :
    : "r0", "r14");
    if (verbose == Z15VER_VERBOSE) {
    printf("getBuffers: historyBuffer: STORAGE OBTAIN rc=%u\\n", rc);
    }
    if (rc != 0) {
    return FALSE;
    }
    }

    return TRUE;
    }

    uint32_t inflate(int checkValue,
    int historyBufferType,
    int bufferLength)
    {
    uint64_t dword_block[192], bytesRead, bytesWritten;
    uint8_t *parameter_block, *op1, *op2, status[32];
    uint32_t buflen, outlen, op1len, op2len, newlen, offset;
    int cc, done, func, ok;
    PDFLTCC_PB0 ppb0;

    /* check that the DFLTCC functions we need are available */
    ok = FALSE;
    if (availableFunctions(status)) {
    if (isAvailable(status, Z15OP_XPND)) {
    ok = TRUE;
    }
    else {
    fprintf(stderr, "ERROR: DFLTCC-XPND function is not available\\n");
    }
    }
    else {
    fprintf(stderr, "ERROR: Unable to retrieve available DFLTCC functions\\n");
    }
    ok = FALSE;
    if (isFormat(status, Z15FMT_PB0)) {
    ok = TRUE;
    }
    else {
    fprintf(stderr, "ERROR: Parameter block format 0 is not available\\n");
    }
    if (!ok) {
    return 0;
    }

    parameter_block = (uint8_t *)(void *)dword_block;
    memset(parameter_block, 0, sizeof(dword_block));
    ppb0 = (PDFLTCC_PB0)(void *)dword_block;
    ppb0->nt = 1;
    if (checkValue) ppb0->cvt = 1; else ppb0->cvt = 0;

    /*
    * outer loop - read block of uncompressed input and deflate them
    */
    done = FALSE;
    bytesRead = 0L;
    bytesWritten = 0L;
    outlen = (bufferLength + Z15BUF_MARGIN) * 1024;
    /* read in first chunk of input file */
    buflen = (int)fread(inBuffer, 1, bufferLength * 1024, fin);
    bytesRead += buflen;
    if (buflen == 0) done = TRUE;
    op1 = outBuffer;
    op1len = outlen;
    op2 = inBuffer;
    op2len = buflen;
    offset = 0;
    while (!done) {
    func = Z15OP_XPND + historyBufferType;
    if (verbose == Z15VER_VERBOSE) {
    printf("inflate: DFLTCC-XPND\\n");
    }
    cc = DFLT(func,
    parameter_block,
    &op1,
    &op1len,
    &op2,
    &op2len,
    historyBuffer);
    if (cc == 0) { /* normal completion */
    if (verbose == Z15VER_VERBOSE) {
    printf("inflate: normal completion\\n");
    }
    if (!writeOut(outBuffer + offset, outlen - op1len - offset, fout, &bytesWritten)) {
    done = TRUE;
    }
    done = TRUE;
    }
    else if (cc == 3) {
    if (verbose == Z15VER_VERBOSE) {
    printf("inflate: partial completion - re-drive DFTLCC\\n");
    }
    /* leave evrything as-is and go around again */
    done = FALSE;
    }
    else if (cc == 2 && ppb0->oesc == 0x00) {
    /* input buffer needs filling, but may not have been emptied */
    if (verbose == Z15VER_VERBOSE) {
    printf("inflate: partial completion - refill input buffer\\n");
    }
    memmove(inBuffer, inBuffer + buflen - op2len, op2len);
    newlen = fread(inBuffer + op2len, 1, buflen - op2len, fin);
    bytesRead += newlen;
    buflen = op2len + newlen;
    op2 = inBuffer;
    op2len = buflen;
    }
    else if (cc == 1) {
    /* output buffer needs emptying */
    if (verbose == Z15VER_VERBOSE) {
    printf("inflate: partial completion - empty output buffer\\n");
    }
    if (!writeOut(outBuffer + offset, outlen - op1len - offset, fout, &bytesWritten)) {
    done = TRUE;
    }
    if (op1len > 0) {
    printf("inflate: *** shouldn't happen - output buffer not full\\n");
    }
    /* preserve required history */
    offset = (historyBufferType == Z15HBT_INLINE) ? ppb0->hl : 0;
    if (offset > 0) {
    memmove(outBuffer, outBuffer + outlen - op1len - offset, offset);
    }
    op1 = outBuffer + offset;
    op1len = outlen - offset;
    }
    else {
    fprintf(stderr, "ERROR occurred during inflation, cc=%u, OESC=x'%02X'\\n",
    cc, ppb0->oesc);
    dumpPB0(ppb0, 384);
    done = TRUE;
    }
    }

    printf("Bytes read from input: %llu\\n", bytesRead);
    printf("Bytes written to output: %llu\\n", bytesWritten);
    if (checkValue == Z15CV_ADLER) {
    printf("Check value (Adler): x'%08X'\\n", ppb0->cv);
    }
    else {
    printf("Check value (CRC-32): x'%08X'\\n", ppb0->cv);
    }

    return ppb0->cv;
    }

    int isAvailable(uint8_t *status, int function_code)
    {
    int result;

    result = FALSE;

    if (function_code < 128 &&
    status[function_code / 8] & (0x80 >> (function_code % 8))) {
    result = TRUE;
    }

    return result;
    }

    int isFormat(uint8_t *status, int format)
    {
    int result;

    result = FALSE;

    if (format < 16 &&
    status[24 + format / 8] & (0x80 >> (format % 8))) {
    result = TRUE;
    }

    return result;
    }

    int openFiles(char *infile, char *outfile, off_t *pinfileSize)
    {
    int rc;
    struct stat st;

    fin = fopen(infile, "rb");
    if (fin == NULL) {
    perror("ERROR: Unable to open input file");
    return FALSE;
    }

    /* we need to know the input file size, because sending DFLTCC
    * zero-length input will cause a specification exception */
    rc = stat(infile, &st);
    if (rc != 0) {
    perror("ERROR: Unable to retrieves statistics for input file");
    return FALSE;
    }
    *pinfileSize = st.st_size;

    fout = fopen(outfile, "wb");
    if (fout == NULL) {
    perror("ERROR: Unable to open output file");
    return FALSE;
    }

    return TRUE;
    }

    double timeDifference(uint64_t t0, uint64_t t1)
    {
    return (((double)(t1 - t0)) * 1.048756 / 4294967296.0);
    }

    int writeOut(uint8_t *buffer, int length, FILE *f, uint64_t *pwritten)
    {
    int rc, result;

    result = TRUE;
    rc = fwrite(buffer, 1, length, f);
    if (rc < length) {
    perror("ERROR: Incomplete write to output file");
    result = FALSE;
    }
    *pwritten += rc;

    return result;
    }

    int DFLT(int function_code,
    uint8_t *parameter_block,
    uint8_t **poperand_1,
    uint32_t *poperand_1_length,
    uint8_t **poperand_2,
    uint32_t *poperand_2_length,
    uint8_t *operand_3)
    {
    uint32_t r0, cc, operand_1_length, operand_2_length;
    uint8_t *operand_1, *operand_2;
    uint64_t t0, t1;

    r0 = (uint32_t)function_code;
    cc = 0;
    operand_1 = *poperand_1;
    operand_1_length = *poperand_1_length;
    operand_2 = *poperand_2;
    operand_2_length = *poperand_2_length;

    if (verbose == Z15VER_VERBOSE) {
    printf(">>> DFLTCC Entry: functionCode=%u\\n", function_code);
    if (function_code == Z15OP_QAF) {
    dump(parameter_block, 32);
    }
    else {
    printf(">>> op1=%08X (Base+%d), op1len=%d (Max-%d)\\n",
    operand_1, (operand_1 == NULL) ? 0 : operand_1 - outBuffer,
    operand_1_length, outMax - operand_1_length);
    printf(">>> op2=%08X (Base+%d), op2len=%d (Max-%d)\\n",
    operand_2, (operand_2 == NULL) ? 0 : operand_2 - inBuffer,
    operand_2_length, inMax - operand_2_length);
    dumpPB0((PDFLTCC_PB0)(void *)parameter_block, (function_code == Z15OP_QAF) ? 32 : 384);
    }
    }

    __asm( " STCKF %[t0] \\n"
    " DFLTCC %[op1],%[op2],%[op3] \\n"
    " IPM %[cc] \\n"
    " SRL %[cc],28 \\n"
    " STCKF %[t1] "
    : [t0] "=m"(t0),
    [t1] "=m"(t1),
    [cc] "=r"(cc),
    [op1] "+RP:r:e"(operand_1),
    "+r"(operand_1_length),
    [op2] "+RP:r:e"(operand_2),
    "+r"(operand_2_length)
    : "NR:r0"(r0),
    "NR:r1"(parameter_block),
    [op3] "r"(operand_3)
    : );

    if (verbose == Z15VER_VERBOSE) {
    printf("<<< DFLTCC Exit: functionCode=%u, cc=%u\\n", function_code, cc);
    if (function_code == Z15OP_QAF) {
    dump(parameter_block, 32);
    }
    else {
    printf("<<< op1=%08X (Base+%d), op1len=%d (Max-%d)\\n",
    operand_1, (operand_1 == NULL) ? 0 : operand_1 - outBuffer,
    operand_1_length, outMax - operand_1_length);
    printf("<<< op2=%08X (Base+%d), op2len=%d (Max-%d)\\n",
    operand_2, (operand_2 == NULL) ? 0 : operand_2 - inBuffer,
    operand_2_length, inMax - operand_2_length);
    dumpPB0((PDFLTCC_PB0)(void *)parameter_block, (function_code == Z15OP_QAF) ? 32 : 384);
    }
    }

    *poperand_1 = operand_1;
    *poperand_1_length = operand_1_length;
    *poperand_2 = operand_2;
    *poperand_2_length = operand_2_length;

    /* update statistics */
    nDfltccCalls++;
    timeInDfltcc += timeDifference(t0, t1);

    return (int)cc;
    }
  8. Compile the program

    The above program can be compiled from z/OS UNIX as follows:

    $ xlc -qasm -qasmlib=sys1.maclib -oz15dflt z15dflt.c
  9. Test the program

    Suppose we have an uncompressed file, uncomp.data, which we want to compress into a new file, comp.def.  To achieve this, we invoke the z15dflt program as follows:

    $ ./z15dflt uncomp.data comp.def
    Bytes read from input: 6803394
    Bytes written to output: 1916628
    Compression ratio: 3.5 : 1
    Check value (CRC-32): x'C2A31B69'
    Calls to DFLTCC: 627
    Time in DFLTCC: 0.002039 s
    Total elapsed time: 0.105293 s

    To reconstitute the original file:

    $ ./z15dflt -i comp.def uncomp.temp
    Bytes read from input: 1916628
    Bytes written to output: 6803394
    Check value (CRC-32): x'C2A31B69'
    Calls to DFLTCC: 263
    Time in DFLTCC: 0.001832 s
    Total elapsed time: 0.006608 s

    To compare the reconstituted file to the original (if the files compare exactly, not output is displayed):

    $ cmp uncomp.data uncomp.temp
    $
  10. A zlib C example

    Here is a program, ztest.c, which is based on the sample program provided at https://zlib.net/zlib_how.html.  This program calls the z/OS-supplied zlib library.  If you invoke this program with the “windowBits” parameter set to -15, it runs in “raw” mode.  In this mode, any compressed files produced by this program should be able to be reconstituted by the above z15dflt program, and vice versa.

    /* ztest.c */
    /*
    * Test IBM zlib library with zEDC present
    */

    #include <assert.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>

    #include "zlib.h"

    /* #define CHUNK 16384 */

    void spitOut(unsigned char *buf, int len)
    {
    int i, j;
    unsigned char txt[16];

    j = 0;
    for (i = 0; i < len; i++) {
    if (j == 0) {
    printf("%04X:", i);
    }
    printf(" %02X", buf[i]);
    if (j == 7) printf(" ");
    if (isprint(buf[i]))
    txt[j] = buf[i];
    else
    txt[j] = '.';
    j++;
    if (j == 16) {
    printf(" >");
    for (j = 0; j < 16; j++) {
    printf("%c", txt[j]);
    }
    printf("<\\n");
    j = 0;
    }
    }
    if (j > 0) {
    for (i = j; i < 16; i++) printf(" ");
    if (j < 7) printf (" ");
    printf(" >");
    for (i = 0; i < j; i++) {
    printf("%c", txt[i]);
    }
    printf("<\\n");
    }
    }

    /* Compress from file source to file dest until EOF on source.
    def() returns Z_OK on success, Z_MEM_ERROR if memory could not be
    allocated for processing, Z_STREAM_ERROR if an invalid compression
    level is supplied, Z_VERSION_ERROR if the version of zlib.h and the
    version of the library linked do not match, or Z_ERRNO if there is
    an error reading or writing the files. */
    int def(FILE *source, FILE *dest, int level, int windowBits, int buflen, unsigned char *in, unsigned char *out)
    {
    int ret, flush;
    unsigned have;
    z_stream strm;

    /* allocate deflate state */
    strm.zalloc = Z_NULL;
    strm.zfree = Z_NULL;
    strm.opaque = Z_NULL;
    ret = deflateInit2(&strm, level, Z_DEFLATED, windowBits, 1, Z_DEFAULT_STRATEGY);
    if (ret != Z_OK)
    return ret;

    /* compress until end of file */
    do {
    strm.avail_in = fread(in, 1, buflen, source);
    if (ferror(source)) {
    (void)deflateEnd(&strm);
    return Z_ERRNO;
    }
    flush = feof(source) ? Z_FINISH : Z_NO_FLUSH;
    strm.next_in = in;

    /* run deflate() on input until output buffer not full, finish
    compression if all of source has been read in */
    do {
    strm.avail_out = buflen;
    strm.next_out = out;
    ret = deflate(&strm, flush); /* no bad return value */
    assert(ret != Z_STREAM_ERROR); /* state not clobbered */
    have = buflen - strm.avail_out;
    if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
    (void)deflateEnd(&strm);
    return Z_ERRNO;
    }
    } while (strm.avail_out == 0);
    assert(strm.avail_in == 0); /* all input will be used */

    /* done when last data in file processed */
    } while (flush != Z_FINISH);
    assert(ret == Z_STREAM_END); /* stream will be complete */

    /* clean up and return */
    (void)deflateEnd(&strm);
    return Z_OK;
    }

    /* Decompress from file source to file dest until stream ends or EOF.
    inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
    allocated for processing, Z_DATA_ERROR if the deflate data is
    invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
    the version of the library linked do not match, or Z_ERRNO if there
    is an error reading or writing the files. */
    int inf(FILE *source, FILE *dest, int windowBits, int buflen, unsigned char *in, unsigned char *out)
    {
    int ret;
    unsigned have;
    z_stream strm;

    /* allocate inflate state */
    strm.zalloc = Z_NULL;
    strm.zfree = Z_NULL;
    strm.opaque = Z_NULL;
    strm.avail_in = 0;
    strm.next_in = Z_NULL;
    ret = inflateInit2(&strm, windowBits);
    if (ret != Z_OK)
    return ret;

    /* decompress until deflate stream ends or end of file */
    do {
    strm.avail_in = fread(in, 1, buflen, source);
    if (ferror(source)) {
    (void)inflateEnd(&strm);
    return Z_ERRNO;
    }
    if (strm.avail_in == 0)
    break;
    strm.next_in = in;

    /* run inflate() on input until output buffer not full */
    do {
    strm.avail_out = buflen;
    strm.next_out = out;
    ret = inflate(&strm, Z_NO_FLUSH);
    assert(ret != Z_STREAM_ERROR); /* state not clobbered */
    switch (ret) {
    case Z_NEED_DICT:
    ret = Z_DATA_ERROR; /* and fall through */
    case Z_DATA_ERROR:
    case Z_MEM_ERROR:
    (void)inflateEnd(&strm);
    return ret;
    }
    have = buflen - strm.avail_out;
    if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
    (void)inflateEnd(&strm);
    return Z_ERRNO;
    }
    } while (strm.avail_out == 0);

    /* done when inflate() says it's done */
    } while (ret != Z_STREAM_END);

    /* clean up and return */
    (void)inflateEnd(&strm);
    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
    }

    /* report a zlib or i/o error */
    void zerr(int ret)
    {
    fputs("ztest: ", stderr);
    switch (ret) {
    case Z_ERRNO:
    if (ferror(stdin))
    fputs("error reading stdin\\n", stderr);
    if (ferror(stdout))
    fputs("error writing stdout\\n", stderr);
    break;
    case Z_STREAM_ERROR:
    fputs("invalid compression level\\n", stderr);
    break;
    case Z_DATA_ERROR:
    fputs("invalid or incomplete deflate data\\n", stderr);
    break;
    case Z_MEM_ERROR:
    fputs("out of memory\\n", stderr);
    break;
    case Z_VERSION_ERROR:
    fputs("zlib version mismatch!\\n", stderr);
    }
    }

    int main(int argc, char **argv)
    {
    int rc, buflen, mode, wb;
    unsigned char *in, *out, *infile, *outfile;
    FILE *source, *dest;

    if (argc != 6) {
    printf("Usage: ztest {d|i} windowBits buflen(KB) infile outfile\\n");
    return 8;
    }
    mode = (int)argv[1][0];
    if (mode != 'd' && mode != 'i') {
    printf("ERROR: Invalid processing mode, '%c'\\n", mode);
    return 8;
    }
    wb = atoi(argv[2]);
    if (wb != -15 && wb != 15 && wb != 31) {
    printf("ERROR: Invalid window bits, %d. Must be -15, 15 or 31\\n", wb);
    return 8;
    }
    buflen = atoi(argv[3]);
    if (buflen < 1) {
    printf("ERROR: Invalid buffer length, %dKB\\n", buflen);
    return 8;
    }
    buflen *= 1024;
    in = malloc(buflen);
    out = malloc(buflen);
    infile = argv[4];
    source = fopen(infile, "rb");
    if (source == NULL) {
    perror("ERROR opening input file");
    return 8;
    }
    outfile = argv[5];
    dest = fopen(outfile, "wb");
    if (dest == NULL) {
    perror("ERROR opening output file");
    fclose(source);
    return 8;
    }

    printf(" \\n");
    printf("Operation: %s\\n", (mode == 'd' ? "DEFLATE" : "INFLATE"));
    printf("Window bits: %d\\n", wb);
    printf("Buffer size: %d bytes\\n", buflen);
    printf("Input file: %s\\n", infile);
    printf("Output file: %s\\n", outfile);

    printf("zlibVersion = %s\\n", zlibVersion());

    rc = deflateHwAvail(buflen);
    printf("deflateHwAvail(%d) = %d\\n", buflen, rc);

    if (mode == 'd') {
    rc = def(source, dest, Z_DEFAULT_COMPRESSION, wb, buflen, in, out);
    }
    else {
    rc = inf(source, dest, wb, buflen, in, out);
    }
    if (rc != Z_OK) {
    zerr(rc);
    }
    fclose(source);
    fclose(dest);
    free(in);
    free(out);

    return rc;
    }

    Compile this program from z/OS UNIX as follows:

    $ xlc -o ztest -I /usr/lpp/hzc/include -L /usr/lpp/hzc/lib ztest.c -lzz

    To compress a file in “raw” mode:

    $ ./ztest d -15 32 uncomp.data comp.def

    Operation: DEFLATE
    Window bits: -15
    Buffer size: 32768 bytes
    Input file: uncomp.data
    Output file: comp.def
    zlibVersion = 1.2.11-zEDC
    deflateHwAvail(32768) = 1

    For more information about calling the zlib API on z/OS, see: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ieac100/UsingZlibForZEnterpriseDataCompress.htm

    Note that z/OS APAR OA56143 is required to enable zlib on z/OS to use the z15 DFLTCC machine instruction.  Also note that several corrective fixes have been supplied for this APAR (see: https://www.ibm.com/support/pages/apar/oa56143)

  11. A Java sample

    Here is a sample Java program, ZipFile.java, which will use the z15 DFLTCC machine instruction, if APAR OA56143 (see above) has been applied to z/OS (Note that Java 8.0 is required, see: https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.80.doc/docs/z_hw_compression.html):

    import java.util.zip.*;
    import java.io.*;

    public class ZipFile {
    public static void main(String[] args) throws IOException {
    String sourceFile = "test1.txt";
    FileOutputStream fos = new FileOutputStream("compressed.zip");
    ZipOutputStream zipOut = new ZipOutputStream(fos);
    File fileToZip = new File(sourceFile);
    FileInputStream fis = new FileInputStream(fileToZip);
    ZipEntry zipEntry = new ZipEntry(fileToZip.getName());
    zipOut.putNextEntry(zipEntry);
    byte[] bytes = new byte[1024];
    int length;
    while((length = fis.read(bytes)) >= 0) {
    zipOut.write(bytes, 0, length);
    }
    zipOut.close();
    fis.close();
    fos.close();
    }
    }

    Here is a corresponding program, UnzipFile.java:

    import java.util.zip.*;
    import java.io.*;

    public class UnzipFile {
    public static void main(String[] args) throws IOException {
    String fileZip = "compressed.zip";
    File destDir = new File("unzip");
    byte[] buffer = new byte[1024];
    ZipInputStream zis = new ZipInputStream(new FileInputStream(fileZip));
    ZipEntry zipEntry = zis.getNextEntry();
    while (zipEntry != null) {
    File newFile = newFile(destDir, zipEntry);
    FileOutputStream fos = new FileOutputStream(newFile);
    int len;
    while ((len = zis.read(buffer)) > 0) {
    fos.write(buffer, 0, len);
    }
    fos.close();
    zipEntry = zis.getNextEntry();
    }
    zis.closeEntry();
    zis.close();
    }

    public static File newFile(File destinationDir, ZipEntry zipEntry) throws IOException {
    File destFile = new File(destinationDir, zipEntry.getName());

    String destDirPath = destinationDir.getCanonicalPath();
    String destFilePath = destFile.getCanonicalPath();

    if (!destFilePath.startsWith(destDirPath + File.separator)) {
    throw new IOException("Entry is outside of the target dir: " + zipEntry.getName());
    }

    return destFile;
    }
    }

Join The Discussion