30 May 2020

30th of June

Farm status
CPU only
Running Rosetta work.

Intel GPUs
All running Rosetta work.

Nvidia GPUs
Two running Rosetta overnight.

Raspberry Pis
Two running Rosetta. Six running Einstein.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
I made it into the top 100 participants for Rosetta. As of this morning I was in 86th place.

The extra network cables arrived so I have used some on the machines that have multiple network cards. They now have two connections into different switches. I also had a 1M cable running between two switches and it was pulling one end so that got replaced by a longer cable.

I had a look around for some simple 2.5GbE switches and managed to only find one from D-Link. Most others have what they call multi-gig ports where it supports 1, 2.5, 5 or 10GbE speeds. The switches I am using at the moment have 8 x 1GbE ports and 2 multi-gig ports, one of which gets used as the up-link. You think there would be something like a 4 x 2.5GbE ports with a high speed up-link, or even an 8 port version.


Farm makeup
At the moment the compute portion of the farm consists of:
Type Number CPU GPU Memory
CPU only 1 AMD Ryzen 3900X (12C/24T) GT710 64GB
Intel GPU 6 Intel i7-8700 (6C/12T) Built-in 32GB
Nvidia GPU 4 AMD Ryzen 3600 (6C/12T) GTX 1660 Ti 32GB

That gives a total of 72 cores/144 threads and 384GB of memory. There is another Ryzen 3900X on order.

23 May 2020

23rd of May

Farm status
CPU only
Running Rosetta work.

Intel GPUs
All running Rosetta work

Nvidia GPUs
Two running Rosetta work

Raspberry Pis
Two running Rosetta. Six running Einstein work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
The 64GB memory kits arrived. I've installed one kit in the CPU only machine and removed the 4 sticks of 16GB that it had. They've been reinstalled into the remaining Nvidia GPU machines. That brings all the 6 core/12 thread machines up to 32GB of memory. The 64GB kits have RGB lighting which I find rather annoying.

The other CPU only machine that I ordered is delayed. The motherboard seems to have been back ordered and keeps changing dates as to when it will arrive.


2.5GbE networking
The 2.5GbE network cards arrived. They only use a PCIe gen3 x1 slot but when I plugged it into an x1 slot the kernel started spamming the console with messages about PCIe bus errors. After swapping it to a free x16 slot it worked fine, so probably something to do with how the PCIe slots are wired up.

The first card went into the proxy server and the second card is in the CPU only machine. I still need to try some speed tests between them and the 10GbE machines (the storage servers).

I used to have the gigabit ports bonded so they share the network traffic but have now switched to running the 2.5 gigabit card as primary and the motherboard (gigabit) port as an active backup.

Debian don't have a driver for the Realtek 8215. Support for the chip was added in August 2019 but the Debian firmware-realtek package is from July 2019. I ended up getting the driver directly from the Linux kernel web site.

I've ordered some more of these EDUP cards off eBay. The original seller increased the price to 69 AUD but I found another seller with them listed in GBP which worked out to 60 AUD each. I only need one more for the next CPU only machine.

Running two network ports per machine now means I need more network cables. Something else to buy.

Update: 26 May 2020
I managed to get one of them to do around 2.5Gbit. According to iperf3 they were doing 2.27Gbit. That was between a 10Gbit and through a few switches into a machine with a 2.5Gbit card. I tested in both directions.

16 May 2020

16th of May

Farm status
CPU only
Running Rosetta work

Intel GPUs
All running Rosetta work

Nvidia GPUs
One running Rosetta work

Raspberry Pis
Two running Rosetta work. Six running Einstein work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
The farm continues to concentrate on Rosetta work on as many cores as possible (within power limitations).


Another Ryzen 3900X build
I ordered parts for another Ryzen 3900X system as they are on sale. Its the same specs as before. The power supply has already arrived. It came from a different supplier. The remaining parts should arrive next week.

The 64GB memory kits (2 x 32GB sticks) came back into stock so I ordered two kits, one for each of the Ryzen 3900X builds. These are from yet another supplier.

The PCH fan on the current Ryzen 3900X system is not spinning. It doesn't seem to be causing any problem at the moment. Its part of the motherboard so I might have to drop the machine off for repair. I would expect the shop would have to replace the motherboard as its soldered on. I will wait until I have built the 2nd Ryzen 3900X before doing this.


2.5GbE networking
I found some EDUP network cards on eBay for 55 AUD each. They have a Realtek chip. I bought two. They are coming from China so I wouldn't be surprised if they fail after a few months or they are really 1GbE network cards. I should be able to get my money back if either happens.

I was thinking of putting one in the proxy server and another in the Ryzen 3900X build. Now that I have a second Ryzen 3900X on order I will probably put one in each of them. I will wait and see how they work before buying any more.

10 May 2020

10th of May

Farm status
CPU only
The Ryzen 3900X is running Rosetta work

Intel GPUs
All running Rosetta work

Nvidia GPUs
Off

Raspberry Pis
Two running Rosetta work. Six running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
Most of last fortnight was taken up with the Ryzen 3900X build, getting it going and doing a burn-in on different projects.

I spent a little time on the proxy server. I put a spare 10GbE network adapter into it which didn't make any noticeable difference. The hardware side was simple, however changing the bonding to adaptive failover took most of my time. There are examples on the internet and most of them don't work any more. Rosetta changed their project comms to HTTPS so that means we cannot cache downloads for them. In the end I went back to the dual 1GbE network adapter that I have used for the last year and bonded the ports.


2.5GbE networking
I did a bit of research into 2.5GbE network adapters. There are a few (mostly based upon Realtek chips). Intel has an i225 chip but it has a bug where the inter-packet gap is incorrect and it requires a respin of the silicon to fix. Aquanta also had one but seem to sell it for more than the 10GbE network adapters.

None of the computer stores are selling them in Australia. There are USB 3 to 2.5GbE external adapters available. I found one PCIe card on Amazon coming from the US. It was 87 AUD. There was confusion over its warranty with Amazon stating it had an "International warranty". Of the 10 reviews two people complained their card failed after 3 months.