examples/CompressedVectorCreate.cpp File Reference

example: creating CompressedVectorNodes from arrays More...

Include dependency graph for CompressedVectorCreate.cpp:

Functions
int	main (int, char **)
	Example use of CompressedVectorNode creation and write/read.

---

Detailed Description

example: creating CompressedVectorNodes from arrays

Also see listing at end of this page for source without line numbers (to cut&paste from).

/*** CompressedVectorCreate.cpp example: creating CompressedVectorNodes from arrays */
#include <iostream>
#include "E57Foundation.h"
using namespace e57;
using namespace std;

struct XYZStruct {
    double x;   
    double y;   
    double z;   
};

int main(int /*argc*/, char** /*argv*/) {
    try {
        ImageFile imf("temp._e57", "w");
        StructureNode root = imf.root();
        
        StructureNode prototype(imf);
        prototype.set("cartesianX", FloatNode(imf));
        prototype.set("cartesianY", FloatNode(imf));
        prototype.set("cartesianZ", FloatNode(imf));
        
        VectorNode codecs(imf, true);

        CompressedVectorNode cv(imf, prototype, codecs);
        root.set("points", cv);

        const int N = 4;
        static double cartesianX[N] = {1.0, 2.0, 3.0, 4.0};
        static double cartesianY[N] = {1.1, 2.1, 3.1, 4.1};
        static double cartesianZ[N] = {1.2, 2.2, 3.2, 4.2};
        vector<SourceDestBuffer> sourceBuffers;
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianX",  cartesianX,  N));
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianY",  cartesianY,  N));
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianZ",  cartesianZ,  N));

        {
            CompressedVectorWriter writer = cv.writer(sourceBuffers);
            writer.write(N);
            writer.close(); // don't forget to explicitly close the CompressedVectorWriter
        }

        imf.close(); // don't forget to explicitly close the ImageFile
    } catch(E57Exception& ex) {
        ex.report(__FILE__, __LINE__, __FUNCTION__);
        return(-1);
    }

    try {
        ImageFile imf("temp._e57", "r");
        StructureNode root = imf.root();

        CompressedVectorNode cv = static_cast<CompressedVectorNode> (root.get("points"));

        const int N = 4;
        XYZStruct cartesianXYZ[N];
        vector<SourceDestBuffer> destBuffers;
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianX", &cartesianXYZ[0].x, N,
                                               false, false, sizeof(XYZStruct)));
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianY", &cartesianXYZ[0].y, N,
                                               false, false, sizeof(XYZStruct)));
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianZ", &cartesianXYZ[0].z, N,
                                               false, false, sizeof(XYZStruct)));

        cout << "CompressedVector has " << cv.childCount() << " child elements" << endl;
        {
            CompressedVectorReader reader = cv.reader(destBuffers);
            
            uint64_t totalRead = 0;
            unsigned nRead = 0;
            while ((nRead = reader.read(destBuffers)) > 0) {
                cout << "Got " << nRead << " points" << endl;
                for (unsigned i = 0; i < nRead; i++) {
                    cout << "point " << totalRead + i << endl;
                    cout << "  cartesianX = " << cartesianXYZ[i].x << endl;
                    cout << "  cartesianY = " << cartesianXYZ[i].y << endl;
                    cout << "  cartesianZ = " << cartesianXYZ[i].z << endl;
                }
                totalRead += nRead;
            }
            reader.close(); // don't forget to explicitly close the CompressedVectorReader
        }

        imf.close(); // don't forget to explicitly close the ImageFile
    } catch(E57Exception& ex) {
        ex.report(__FILE__, __LINE__, __FUNCTION__);
        return(-1);
    }
    return(0);
}

This example program writes a CompressedVectorNode in an ImageFile from three separate arrays, each containing 4 double coordinates. The CompressedVectorNode is then read back into a single array of structures. See the HelloWorld.cpp example for discussion of the use of include files, constructing an ImageFile, and the try/catch block to handle exceptions.

In source lines 20-21, a new ImageFile is opened for writing, and its predefined root node is fetched. In source lines 23-26, a prototype structure is created that describes the fields that will be stored in each record in the CompressedVectorNode. Each record will consist of 3 double precision floating-point numbers, with element names cartesianX, cartesianY, and cartesianZ.

Source line 28 creates an empty heterogeneous VectorNode whose handle is stored in variable name codecs. A codec (coder/decoder) is a pair of algorithms that copy the data between main memory and the data file. The coder/decoder algorithms are implemented internally in an E57 Foundation Implementation. The coder is utilized during the writing of an E57 file, and the decoder is utilized during the reading of an E57 file. A coder algorithm gets a data item (a double in this example) from a memory buffer in the writing program, perhaps does some processing on it to make it smaller, and then stores the result in the disk file. A decoder algorithm gets some data from the previously written disk file, undoes any processing that the coder did, and stores the reconstituted data into a memory buffer in the reading program. Currently there is only one codec option (bitPackCodec) in the Reference Implementation, and it is the default option if no codecs are specified for a field in the record. So an empty codecs VectorNode requests the bitPackCodec for each of the three fields in the record.

Technically, there are four elements in the prototype tree: three FloatNodes and a StructureNode which contains them. However, container elements in the prototype tree do not need to be encoded in the binary section of the E57 file, so they don't need a codec specified for them. The values stored in the three FloatNodes are all zero (that's what the default to in the constructors on source lines 24-26). In a prototype, however, the values are ignored. It is the type of the node and any specified limits on what value can be stored that are important. In the example, the type is FloatNode, the precision is E57_DOUBLE, and the default limits are set to the smallest and largest values possible in a double precision float (so effectively there are no limits to the value that can be stored). Because no compression is used in the coder, each record will require 3*64=192 bits of storage in the file.

source line 30 creates the CompressedVectorNode using the two trees prototype and codecs. The CompressedVectorNode cv is attached into the tree of the ImageFile, under the path name "/points". The two trees prototype and codecs don't contain any data values and aren't connected into the data tree of an ImageFile in the same way as nodes that do contain values. They function more like the arguments to the constructors of other Node types (e.g. the precision of FloatNode, or the byteCount of BlobNode). They don't get path names. However you can get them back by fetching the CompressedVectorNode by its path name, then call CompressedVectorNode::prototype or CompressedVectorNode::codecs.

Three built-in C++ arrays are specified in source lines 34-36, each holding four coordinates. Although it doesn't matter in this example, the arrays are declared static, which means they are not stored on the stack. Storing large built-in arrays on the stack risks causing a stack overflow. In source lines 37-40, a SourceDestBuffer object is created that describes each array, and a vector of the three SourceDestBuffers is created. The vector function push_back appends the SourceDestBuffers one-at-a-time to the vector. Constructing a three element vector and assigning the SourceDestBuffer to each element (e.g. sourceBuffers[0] = SourceDestBuffer(...)) won't work because SourceDestBuffer has no default constructor (it is impossible to make the equivalent of a NULL handle). In the construction of each SourceDestBuffer, the address of the buffer is given, along with the number of elements in the buffer, and the pathname in the prototype that identifies the field of the record that will get the values stored in the buffer. For writers, all the fields specified in the prototype must be written at one time. The writer cannot write each field separately. Readers may read any combination of defined fields that suit them.

The types of memory buffers for both writer and reader in this example program match the representation specified in the prototype. Therefore no representation conversions will be required during writing or reading. If there were conversions needed, they would be specified with additional arguments during the construction of the SourceDestBuffers.

The local block in source lines 42-46 will create an iterator that can write blocks of data into the CompressedVector. In this example only a single block is written. A C++ local block (the nested {}) is used to limit the scope of the iterator variable writer. It is good practice to control the scope of reader/writer iterators, so that you can control their lifetimes. In this example, the local block is not essential.

On source line 43, the CompressedVectorWriter iterator object is created by specifying which buffers the iterator will transfer data from. The creation of the CompressedVectorWriter doesn't perform the first write. The first write is requested explicitly as in source line 44. In programming, iterators often process one item in a collection at a time. In the Foundation API, however, for efficiency reasons the data should be processed in blocks of 100s or 1000s of points at a time. The specified buffers can be refilled after they are written, and the iterator can be called for a second write with just the number of points to write (in source line 44), or a different buffers can be filled and handed to the iterator using an overloaded write function that takes a new vector of SourceDestBuffers.

In source line 45, the CompressedVectorWriter::close must be explicitly to signal the end of the write. This call is required for much the same reason as the ImageFile::close is required: communicating error conditions from a destructor using exceptions is impossible. If CompressedVectorWriter::close hadn't been called, the points written using the CompressedVectorWriter would have been discarded.

In source lines 55-56, the E57 file is reopened and the root node is fetched. In source line 58 the CompressedVectorNode is fetched by name, and downcast to the type that it is known to be (without checking). Since this program just wrote the file, the reader routine knows that the "/points" CompressedVectorNode exists and that the prototype has three elements in it that store double precision floating point numbers. Readers that have no guarantee of the layout of the file must consult the prototype to determine what fields they recognize and what memory representation types are appropriate. Often the number of possibilities are constrained by the ASTM E57 format standard or by the documentation of an extension.

In source lines 61-68, the SourceDestBuffer objects that describe the memory buffers that will receive the data are created. In contrast to the writer, the organization of the XYZ coordinates is different in the reader routine. The XYZ values are no longer stored in separate arrays. They are stored as a single array of XYZStruct. This is handled by using the stride argument in the SourceDestBuffer constructor. Three SourceDestBuffers are still needed, but stride value is set to the size of the XYZStruct. When the CompressedVectorReader iterator writes a series of x values into the buffer, instead of storing them contiguously, the iterator advances by adding the stride value to the output pointer, thereby skipping over the intervening yz fields to the next x field. Note that in source line 63 the address of the buffer passed to the SourceDestBuffer constructor is not the start of the whole array, but the address of the first x (the same is true for the y, and z fields).

In source line 72, the CompressedVectorReader iterator is created. Unlike the writer case, the reader does not have to request all the defined fields in the read. In the while loop on source lines 76-83, blocks of records are read until the read function returns 0. Alternatively, the number of records defined could have been fetched using CompressedVectorNode::childCount. Like the CompressedVectorWriter, the CompressedVectorReader is explicitly closed in source line 86.

Note that the FloatNode values written in the file do not appear in the XML listing, as they are stored in the binary section of the file that is not printed by the utility (E57xmldump.exe) that generated the listing. On XML line 4, you can see the fileOffset XML attribute that indicates the binary section that holds the CompressedVectorNode values starts at the physical offset of 40 (decimal) in the disk file.

The following console output is produced:

The XML section of the temp._e57 E57 file produced by this example program is as follows:

Here is the source code without line numbers to cut&paste from:

/*** CompressedVectorCreate.cpp example: creating CompressedVectorNodes from arrays */
#include <iostream>
#include "E57Foundation.h"
using namespace e57;
using namespace std;

struct XYZStruct {
    double x;   
    double y;   
    double z;   
};

int main(int /*argc*/, char** /*argv*/) {
    try {
        ImageFile imf("temp._e57", "w");
        StructureNode root = imf.root();
        
        StructureNode prototype(imf);
        prototype.set("cartesianX", FloatNode(imf));
        prototype.set("cartesianY", FloatNode(imf));
        prototype.set("cartesianZ", FloatNode(imf));
        
        VectorNode codecs(imf, true);

        CompressedVectorNode cv(imf, prototype, codecs);
        root.set("points", cv);

        const int N = 4;
        static double cartesianX[N] = {1.0, 2.0, 3.0, 4.0};
        static double cartesianY[N] = {1.1, 2.1, 3.1, 4.1};
        static double cartesianZ[N] = {1.2, 2.2, 3.2, 4.2};
        vector<SourceDestBuffer> sourceBuffers;
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianX",  cartesianX,  N));
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianY",  cartesianY,  N));
        sourceBuffers.push_back(SourceDestBuffer(imf, "/cartesianZ",  cartesianZ,  N));

        {
            CompressedVectorWriter writer = cv.writer(sourceBuffers);
            writer.write(N);
            writer.close(); // don't forget to explicitly close the CompressedVectorWriter
        }

        imf.close(); // don't forget to explicitly close the ImageFile
    } catch(E57Exception& ex) {
        ex.report(__FILE__, __LINE__, __FUNCTION__);
        return(-1);
    }

    try {
        ImageFile imf("temp._e57", "r");
        StructureNode root = imf.root();

        CompressedVectorNode cv = static_cast<CompressedVectorNode> (root.get("points"));

        const int N = 4;
        XYZStruct cartesianXYZ[N];
        vector<SourceDestBuffer> destBuffers;
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianX", &cartesianXYZ[0].x, N,
                                               false, false, sizeof(XYZStruct)));
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianY", &cartesianXYZ[0].y, N,
                                               false, false, sizeof(XYZStruct)));
        destBuffers.push_back(SourceDestBuffer(imf, "/cartesianZ", &cartesianXYZ[0].z, N,
                                               false, false, sizeof(XYZStruct)));

        cout << "CompressedVector has " << cv.childCount() << " child elements" << endl;
        {
            CompressedVectorReader reader = cv.reader(destBuffers);
            
            uint64_t totalRead = 0;
            unsigned nRead = 0;
            while ((nRead = reader.read(destBuffers)) > 0) {
                cout << "Got " << nRead << " points" << endl;
                for (unsigned i = 0; i < nRead; i++) {
                    cout << "point " << totalRead + i << endl;
                    cout << "  cartesianX = " << cartesianXYZ[i].x << endl;
                    cout << "  cartesianY = " << cartesianXYZ[i].y << endl;
                    cout << "  cartesianZ = " << cartesianXYZ[i].z << endl;
                }
                totalRead += nRead;
            }
            reader.close(); // don't forget to explicitly close the CompressedVectorReader
        }

        imf.close(); // don't forget to explicitly close the ImageFile
    } catch(E57Exception& ex) {
        ex.report(__FILE__, __LINE__, __FUNCTION__);
        return(-1);
    }
    return(0);
}