一.MNIST手写字体文件说明
MNIST手写字体数据库下载地址http://yann.lecun.com/exdb/mnist/ 。
MNIST手写字体的数据库说明在下载网站的下面也有,为了便于写程序,数据库文件说明如下:
从上面的数据库说明可以看出来,MNIST手写字体数据库包含了是个文件,每个文件都是单纯的普通文件格式,因此,可以采用C++的文件流将其打开,每一个文件除了几个字节的文件头之外,就是剩下的要数据部分了。因此,可以先将文件的文件头读进来,然后利用magic number进行验证,验证所读的文件是否为MNIST文件。
由于MNIST存储的格式是大端存储的,和大部分Intel处理器的存储方式不同,所以,直接将文件头的前面四个直接存储为int类型或者long类型的话,是无法获得正确的数值的,还需要进行从大端模式到小端模式的转换,而且,在不同的处理器上面,int类型和long类型存储的位数是不一样的,所以,用int类型或者long类型来读取文件头的前面四个字节不具有可移植性。
在C++标准中,char类型的长度被定义为一个字节,这个在不同的处理器上面是不变的,因此,可以采用char类型的数组来存储文件头的部分,同时,使用char类型的数组来进行大端模式到小端模式的转换也是很容易的。
大端模式和小端模式的定义的区别如下:
大端模式:高位字节放在内存低地址处,低位字节放在内存高地址处;
小端模式:低位字节放在内存低地址处,高位字节放在内存高地址处;Intel处理器一般为小端模式。
下面是定义图片文件的文件头和标签的文件头
struct MNISTImageFileHeader { unsigned char MagicNumber[4]; unsigned char NumberOfImages[4]; unsigned char NumberOfRows[4]; unsigned char NumberOfColums[4]; };
struct MNISTLabelFileHeader { unsigned char MagicNumber[4]; unsigned char NumberOfLabels[4]; };
由于大端模式是把高位字节放在内存的低位处,所以,char类型的数组的低字节表示的就是原来的数的高位,所以,在将char类型的数组转化为整数的时候,只要将数组的第一个元素左移24位,第二个元素左移16位,第三个元素左移8位,然后将这些元素相加起来就可以了,下面是用迭代的方法实现的代码。
int ConvertCharArrayToInt(unsigned char* array, int LengthOfArray) { if (LengthOfArray < 0) { return -1; } int result = static_cast<signed int>(array[0]); for (int i = 1; i < LengthOfArray; i++) { result = (result << 8) + array[i]; } return result; }
在有了上面的将大端模式存储的char类型数组转化为整数之后,我们就可以将MNIST的文件按照普通的文件格式读进来了,在读取了文件头之后,就可以用普通的读文件的方法将数据部分放在一个char数组中,然后就可以将其放在OpenCV的Mat对象中了。下面是全部读取MNIST文件的代码。
#ifndef MNIST_H #define MNIST_H #include <iostream> #include <fstream> #include <opencv2/opencv.hpp> struct MNISTImageFileHeader { unsigned char MagicNumber[4]; unsigned char NumberOfImages[4]; unsigned char NumberOfRows[4]; unsigned char NumberOfColums[4]; }; struct MNISTLabelFileHeader { unsigned char MagicNumber[4]; unsigned char NumberOfLabels[4]; }; const int MAGICNUMBEROFIMAGE = 2051; const int MAGICNUMBEROFLABEL = 2049; int ConvertCharArrayToInt(unsigned char* array, int LengthOfArray); bool IsImageDataFile(unsigned char* MagicNumber, int LengthOfArray); bool IsLabelDataFile(unsigned char* MagicNumber, int LengthOfArray); cv::Mat ReadData(std::fstream& DataFile, int NumberOfData, int DataSizeInBytes); cv::Mat ReadImageData(std::fstream& ImageDataFile, int NumberOfImages); cv::Mat ReadLabelData(std::fstream& LabelDataFile, int NumberOfLabel); cv::Mat ReadImages(std::string& FileName); cv::Mat ReadLabels(std::string& FileName); #endif // MNIST_H
/** * @file ReadData.cpp The file contains the functions used to read image data * and label data from the origin mnist file * @author sheng * @version 1.0.0 * @date 2014-04-09 * * @function * * @histroy <author> <date> <version> <description> * sheng 2014-04-09 1.0.0 build the module */ #include <MNIST.h> /** * @brief IsImageDataFile Check the input MagicNumber is equal to * MAGICNUMBEROFIMAGE * @param MagicNumber The array of the magicnumber to be checked * @param LengthOfArray The length of the array * @return true, if the magcinumber is mathed; * false, otherwise. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ bool IsImageDataFile(unsigned char* MagicNumber, int LengthOfArray) { int MagicNumberOfImage = ConvertCharArrayToInt(MagicNumber, LengthOfArray); if (MagicNumberOfImage == MAGICNUMBEROFIMAGE) { return true; } return false; } /** * @brief IsImageDataFile Check the input MagicNumber is equal to * MAGICNUMBEROFLABEL * @param MagicNumber The array of the magicnumber to be checked * @param LengthOfArray The length of the array * @return true, if the magcinumber is mathed; * false, otherwise. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ bool IsLabelDataFile(unsigned char *MagicNumber, int LengthOfArray) { int MagicNumberOfLabel = ConvertCharArrayToInt(MagicNumber, LengthOfArray); if (MagicNumberOfLabel == MAGICNUMBEROFLABEL) { return true; } return false; } /** * @brief ReadData Read the data in a opened file * @param DataFile The file which the data is read from. * @param NumberOfData The number of the data * @param DataSizeInBytes The size fo the every data * @return The Mat which rows is a data, * Return a empty Mat if the file is not opened or the some flag was * seted when reading the data. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ cv::Mat ReadData(std::fstream& DataFile, int NumberOfData, int DataSizeInBytes) { cv::Mat DataMat; // read the data if the file is opened. if (DataFile.is_open()) { int AllDataSizeInBytes = DataSizeInBytes * NumberOfData; unsigned char* TmpData = new unsigned char[AllDataSizeInBytes]; DataFile.read((char *)TmpData, AllDataSizeInBytes); // // If the state is good, convert the array to a mat. // if (!DataFile.fail()) // { // DataMat = cv::Mat(NumberOfData, DataSizeInBytes, CV_8UC1, // TmpData).clone(); // } DataMat = cv::Mat(NumberOfData, DataSizeInBytes, CV_8UC1, TmpData).clone(); delete [] TmpData; DataFile.close(); } return DataMat; } /** * @brief ReadImageData Read the Image data from the MNIST file. * @param ImageDataFile The file which contains the Images. * @param NumberOfImages The number of the images. * @return The mat contains the image and each row of the mat is a image. * Return empty mat is the file is closed or the data is not matching * the number. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ cv::Mat ReadImageData(std::fstream& ImageDataFile, int NumberOfImages) { int ImageSizeInBytes = 28 * 28; return ReadData(ImageDataFile, NumberOfImages, ImageSizeInBytes); } /** * @brief ReadLabelData Read the label data from the MNIST file. * @param LabelDataFile The file contained the labels. * @param NumberOfLabel The number of the labels. * @return The mat contains the labels and each row of the mat is a label. * Return empty mat is the file is closed or the data is not matching * the number. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ cv::Mat ReadLabelData(std::fstream& LabelDataFile, int NumberOfLabel) { int LabelSizeInBytes = 1; return ReadData(LabelDataFile, NumberOfLabel, LabelSizeInBytes); } /** * @brief ReadImages Read the Training images. * @param FileName The name of the file. * @return The mat contains the image and each row of the mat is a image. * Return empty mat is the file is closed or the data is not matched. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ cv::Mat ReadImages(std::string& FileName) { std::fstream File(FileName.c_str(), std::ios_base::in | std::ios_base::binary); if (!File.is_open()) { return cv::Mat(); } MNISTImageFileHeader FileHeader; File.read((char *)(&FileHeader), sizeof(FileHeader)); if (!IsImageDataFile(FileHeader.MagicNumber, 4)) { return cv::Mat(); } int NumberOfImage = ConvertCharArrayToInt(FileHeader.NumberOfImages, 4); return ReadImageData(File, NumberOfImage); } /** * @brief ReadLabels Read the label from the MNIST file. * @param FileName The name of the file. * @return The mat contains the image and each row of the mat is a image. * Return empty mat is the file is closed or the data is not matched. * * @author sheng * @version 1.0.0 * @date 2014-04-08 * * @histroy <author> <date> <version> <description> * sheng 2014-04-08 1.0.0 build the function */ cv::Mat ReadLabels(std::string& FileName) { std::fstream File(FileName.c_str(), std::ios_base::in | std::ios_base::binary); if (!File.is_open()) { return cv::Mat(); } MNISTLabelFileHeader FileHeader; File.read((char *)(&FileHeader), sizeof(FileHeader)); if (!IsLabelDataFile(FileHeader.MagicNumber, 4)) { return cv::Mat(); } int NumberOfImage = ConvertCharArrayToInt(FileHeader.NumberOfLabels, 4); return ReadLabelData(File, NumberOfImage); }