At last, some coherent results using Nvidia docker container nvcr.io/nvidia/digits:18.11-caffe :

The main thing to look at is the mAP (val) and this peaked at about 21%. Now I can try adding a few more images and see how they affect the accuracy :)