Getting the hardware running

A project log for 32-TFLOP Deep Learning GPU Box

A super-fast linux-based machine with multiple GPUs for training deep neural nets

robotbugsrobotbugs 06/04/2016 at 04:330 Comments

The building process was not too hard and kind of fun. The components look impressive.

One hassle with putting this together was fitting the water cooling radiator to the provided fan and attaching this to the case. I had to tie the thing in place with zip ties so that I could then put the bolts through and insert the washers without it all falling apart and losing small parts in the case.

It's not obvious which holes to use for screwing the PSU to the wall of the case. Some holes that look like holes are not threaded so it was easy to cross thread the screws into these - inspect carefully which are legitimate holes.

The mains cabe is very thick. With 1600W the machine basically needs its own 15A circuit. However the four GTX 1080s are not too power hungry so I think the PSU is probably over specified.

Another hassle was trying to plug all the cables into the motherboard when its actually quite dark in there (I used a flashlight) and also it was easy to bend the fine pins on the USB board interconnects.

There are four fans and a water pump in the cooling system. I plugged the fan that is on the radiator into the CPU_FAN board connector and the water pump into the CPU_OPT connector. There are three other fan connectors distributed around the board which I used for the case fans.

It's important to put the RAM in the right location. This system uses 4 DDR4 modules to make up 32GB. These go in the gray RAM sockets.

I was pleased to find that it booted up fine.

I upgraded the BIOS. One can use the QFLASH tool with a USB drive that contained the BIOS from the Gigabyte site. Initially I was getting an error trying to upgrade but eventually I realized that I was using the X99-SLI BIOS instead of the one for X99P-SLI.

The big problem that stopped me for almost a week was that the BIOS could not see the M.2 drive, so I could not install any OS. I looked everywhere for solutions on line and throughout every setting on the BIOS. In the end I replaced both the M.2 drive and the motherboard and then it worked ok. So I think there was some problem with the motherboard. Note that this motherboard does not support SATA M.2 drives, they must be PCIe.