Details about tiny-dnn's API and short examples.
construct the network model
There are two types of network model available: sequential and graph. A graph representation describe network as computational graph - each node of graph is layer, and each directed edge holds tensor and its gradients. Sequential representation describe network as linked list - each layer has at most one predecessor and one successor layer. Two types of network is represented as network<sequential> and network<graph> class. These two classes have same API, except for its construction.
sequential model
You can construct networks by chaining operator <<
from top(input) to bottom(output).
network<sequential> net;
net << convolutional_layer<tan_h>(32, 32, 5, 1, 6)
<< average_pooling_layer<tan_h>(28, 28, 6, 2)
<< fully_connected_layer<tan_h>(14 * 14 * 6, 120)
<< fully_connected_layer<identity>(120, 10);
network<sequential> net;
net << convolutional_layer<relu>(32, 32, 5, 3, 9)
<< average_pooling_layer<relu>(28, 28, 9, 2)
<< fully_connected_layer<tan_h>(14 * 14 * 9, 120)
<< fully_connected_layer<softmax>(120, 40);
If you feel these syntax a bit redundant, you can also use "shortcut" names defined in tiny_dnn.h.
using namespace tiny_dnn::layers;
Simple image utility class.
Definition image.h:94
If your network is simple mlp(multi-layer perceptron), you can also use make_mlp
function.
It is equivalent to:
graph model
To construct network which has branch/merge in their model, you can use network<graph>
class. In graph model, you should declare each "node" (layer) at first, and then connect them by operator <<
. If two or more nodes are fed into 1 node, operator,
can be used for this purpose. After connecting all layers, call construct_graph
function to register node connections to graph-network.
construct_graph(net, { &
in1, &
in2 }, { &out });
element-wise add N vectors y_i = x0_i + x1_i + ... + xnum_i
Definition arithmetic_layer.h:36
train the model
regression
Use network::fit
function to train. Specify loss function by template parameter (mse
, cross_entropy
, cross_entropy_multiclass
are available), and fed optimizing algorithm into first argument.
std::vector<vec_t>
input_data { { 1, 0 }, { 0, 2 } };
Definition loss_function.h:33
bool fit(Optimizer &optimizer, const std::vector< T > &inputs, const std::vector< U > &desired_outputs, size_t batch_size, int epoch, OnBatchEnumerate on_batch_enumerate, OnEpochEnumerate on_epoch_enumerate, const bool reset_weights=false, const int n_threads=CNN_TASK_SIZE, const std::vector< U > &t_cost=std::vector< U >())
trains the network for a fixed number of epochs to generate desired output.
Definition network.h:312
adaptive gradient method
Definition optimizer.h:77
If you want to do something for each epoch / minibatch (profiling, evaluating accuracy, saving networks, changing learning rate of optimizer, etc), you can register callbacks for this purpose.
classification
As with regression task, you can use network::fit
function in classification. Besides, if you have labels(class-id) for each training data, network::train
can be used. Difference between network::fit
and network::train
is how to specify the desired outputs - network::train
takes label_t
type, instead of vec_t
.
std::vector<vec_t>
input_data { { 1, 0 }, { 0, 2 } };
bool train(Optimizer &optimizer, const std::vector< vec_t > &inputs, const std::vector< label_t > &class_labels, size_t batch_size, int epoch, OnBatchEnumerate on_batch_enumerate, OnEpochEnumerate on_epoch_enumerate, const bool reset_weights=false, const int n_threads=CNN_TASK_SIZE, const std::vector< vec_t > &t_cost=std::vector< vec_t >())
trains the network for a fixed number of epochs (for classification task)
Definition network.h:247
train graph model
If you train graph network, be sure to fed input/output data which has same shape to network's input/output layers.
construct_graph(net, { &
in1, &
in2 }, { &
fc });
std::vector<tensor_t> data{ { { 1, 0 }, { 3, 2 } },{ { 0, 2 }, { 1, 1 } } };
std::vector<tensor_t> out { { { 2, 5 } },{ { 3, 1 } } };
concat N layers along depth
Definition concat_layer.h:44
without callback
...
Definition loss_function.h:127
base class of optimizer usesHessian : true if an optimizer uses hessian (2nd order derivative of loss...
Definition optimizer.h:37
with callback
"freeze" layers
You can use layer::set_trainable
to exclude a layer from updating its weights.
net[1]->set_trainable(false);
use/evaluate trained model
predict a value
vec_t in = {1.0, 2.0, 3.0};
vec_t predict(const vec_t &in)
executes forward-propagation and returns output
Definition network.h:187
double in[] = {1.0, 2.0, 3.0};
caclulates output vector for given input. You can use vec_t
, std::vector<float>
, double[]
and any other range as input.
We also provide predict_label
and predict_max_value
for classification task.
}
label_t predict_label(const vec_t &in)
executes forward-propagation and returns maximum output index
Definition network.h:209
float_t predict_max_value(const vec_t &in)
executes forward-propagation and returns maximum output
Definition network.h:202
evaluate accuracy
caluculate the loss
float_t get_loss(const std::vector< vec_t > &in, const std::vector< vec_t > &t)
calculate loss value (the smaller, the better) for regression task
Definition network.h:421
You must specify loss-function by template parameter. We recommend you to use the same loss-function to training.
visualize the model
visualize graph networks
We can get graph structure in dot language format.
std::ofstream
ofs(
"graph_net_example.txt");
utility for graph visualization
Definition graph_visualizer.h:38
Once we get dot language model, we can easily get an image by graphviz:
dot -Tgif graph_net_example.txt -o graph.gif
Then you can get:

visualize each layer activations
...
image
img =
nn[0]->output_to_image();
visualize convolution kernels
...
img.write(
"kernel0.bmp");
io
save and load the model
You can use network::save
and network::load
to save/load your model:
...
nn.save("my-network");
The generated binary file will contain:
- the architecture of model
- the weights of the model
You can also select file format, and what you want to save:
nn.save(
"my-weights", content_type::weights, file_format::binary);
nn.load(
"my-weights", content_type::weights, file_format::, file_format::json););
nn.save(
"my-architecture", content_type::model, file_format::json);
nn.load(
"my-architecture", content_type::model, file_format::json);
nn.save(
"my-network", content_type::weights_and_model, file_format::binary);
nn.load(
"my-network", content_type::weights_and_model, file_format::binary);
If you want the architecture model in string
format, you can use to_json
and from_json
.
std::string json =
nn.to_json();
Note: operator <<
and operator >>
APIs before tiny-dnn v0.1.1 are deprecated.
import caffe's model
Import Caffe Model to tiny-dnn
reading data
from MNIST idx format
parse_mnist_images(
"train-images.idx3-ubyte", &
images, -1.0, 1.0, 2, 2);
parse_mnist_labels("train-labels.idx1-ubyte", &labels);
from cifar-10 binary format
parse_cifar10(
"data_batch1.bin", &
images, &labels, -1.0, 1.0, 0, 0);
reading images
You can use a simple tiny_dnn::image
class to handle your images. JPEG (baseline & progressive), PNG (1/2/4/8 bit per channel), BMP (non-1bp, non-RLE), GIF are supported reading formats. Note that it's memory layout differs from OpenCV - it's layout is KHW (K:channels, H:height, W:width).
std::vector<uint8_t> rgb =
rgb_img.to_rgb();
rgb_img.from_rgb(rgb.begin(), rgb.end());
get/set the properties
traverse layers
base class of all kind of NN layers
Definition layer.h:62
std::cout <<
l->layer_type() << std::endl;
}
const T & at(size_t index) const
return index-th layer as <T> throw nn_error if index-th layer cannot be converted to T
Definition network.h:539
graph_traverse(net[0],
std::cout <<
l.layer_type() << std::endl;
},
std::cout <<
e.vtype() << std::endl;
});
class containing input/output data
Definition node.h:95
get layer types
You can access each layer by operator[] after construction.
...
for (
int i = 0;
i <
nn.depth();
i++) {
cout <<
"#layer:" <<
i <<
"\n";
cout <<
"layer type:" <<
nn[
i]->layer_type() <<
"\n";
cout <<
"input:" <<
nn[
i]->in_data_size() <<
"(" <<
nn[
i]->in_data_shape() <<
")\n";
cout <<
"output:" <<
nn[
i]->out_data_size() <<
"(" <<
nn[
i]->out_data_shape() <<
")\n";
}
single-input, single-output feedforward network
Definition nodes.h:272
output:
#layer:0
layer type:conv
input:3072([[32x32x3]])
output:4704([[28x28x6]])
num of parameters:456
#layer:1
layer type:max-pool
input:4704([[28x28x6]])
output:1176([[14x14x6]])
num of parameters:0
#layer:2
layer type:fully-connected
input:1176([[1176x1x1]])
output:10([[10x1x1]])
num of parameters:11770
get weight vector
std::vector<vec_t*> weights =
nn[
i]->get_weights();
Number of elements differs by layer types and settings. For example, in fully-connected layer with bias term, weights[0] represents weight matrix and weights[1] represents bias vector.
change the weight initialization
In neural network training, initial value of weight/bias can affect training speed and accuracy. In tiny-dnn, the weight is appropriately scaled by xavier algorithm1 and the bias is filled with 0.
To change initialization method (or weight-filler) and scaling factor, use weight_init()
and bias_init()
function of network and layer class.
- xavier ... automatic scaling using sqrt(scale / (fan-in + fan-out))
- lecun ... automatic scaling using scale / sqrt(fan-in)
- constant ... fill constant value
Definition weight_init.h:102
Use fan-in(number of input weight for each neuron) for scaling.
Definition weight_init.h:75
Use fan-in and fan-out for scaling.
Definition weight_init.h:56
change the seed value
You can change the seed value for the random value generator.
Note: Random value generator is shared among thread.
tune the performance
profile
change the number of threads while training
macro defines the number of threads for parallel training. Change it to smaller value will reduce memory footprint. This change affects execution time of training the network, but no affects on prediction.
handle errors
When some error occurs, tiny-dnn doesn't print any message on stdout. Instead of printf
, tiny-dnn throws exception. This behaviour is suitable when you integrate tiny-dnn into your application (especially embedded systems).
catch application exceptions
tiny-dnn may throw one of the following types:
is derived from nn_error
, and they have what()
method to provide detail message about the error.
try {
...
}
error exception class for tiny-dnn
Definition nn_error.h:37