Farm status
CPU only
Running Einstein work
Intel GPUs
Off
Nvidia GPUs
Off
Raspberry Pis
Running Einstein work
Farm news
Another hot week so not much crunching done.
Rosetta has some issue where it doesn't have any work to send despite the server status page saying there are 11 million work units ready. In the mean time everything that is running is doing Einstein work.
28 December 2020
19 December 2020
19th of December
Farm status
CPU only
Running Rosetta
Intel GPUs
Four running Rosetta
Nvidia GPUs
Off
Raspberry Pis
Six running Einstein. Three running Einstein and Rosetta (50/50 split).
Other news
Its been a hot and humid week so everything except the Pis have been off. Today is a somewhat cooler 24 degrees C with drizzle on and off (ie 100% humidity). I've fired up a few of the Intel GPU machines and have them doing Rosetta.
I am still waiting for the 2nd Ryzen 5900X to be shipped. The supplier status is "We are expecting a small shipment this week. Once we have confirmed numbers, we will update with an expected new queue position. Queue is just shy of 50% filled since launch". I ordered the second 5900X on the 6th of November so its been six weeks so far. Maybe Santa will bring a new CPU for christmas.
Next year plans
I was thinking of rationalizing the farm a bit and bringing it up to 4 x Ryzen 5900X machines and get rid of the Intel GPU machines completely (6 x i7-8700). That reduces the physical machines while not losing too many CPU cores.
Another area that I would like to look at is an AMD GPU machine or two. Currently there are 4 x Ryzen 5600 with Nvidia GTX 1660Ti in the farm. I haven't used AMD GPUs before due to issues with their drivers under Linux.
Ampere Altra
They are now shipping. Some reviews have started to surface. Apparently the 80 core 3.3Ghz model is faster than an AMD Threadripper at compute performance. I made inquiries about getting one but never heard back from my server supplier. I will have to ask again.
CPU only
Running Rosetta
Intel GPUs
Four running Rosetta
Nvidia GPUs
Off
Raspberry Pis
Six running Einstein. Three running Einstein and Rosetta (50/50 split).
Other news
Its been a hot and humid week so everything except the Pis have been off. Today is a somewhat cooler 24 degrees C with drizzle on and off (ie 100% humidity). I've fired up a few of the Intel GPU machines and have them doing Rosetta.
I am still waiting for the 2nd Ryzen 5900X to be shipped. The supplier status is "We are expecting a small shipment this week. Once we have confirmed numbers, we will update with an expected new queue position. Queue is just shy of 50% filled since launch". I ordered the second 5900X on the 6th of November so its been six weeks so far. Maybe Santa will bring a new CPU for christmas.
Next year plans
I was thinking of rationalizing the farm a bit and bringing it up to 4 x Ryzen 5900X machines and get rid of the Intel GPU machines completely (6 x i7-8700). That reduces the physical machines while not losing too many CPU cores.
Another area that I would like to look at is an AMD GPU machine or two. Currently there are 4 x Ryzen 5600 with Nvidia GTX 1660Ti in the farm. I haven't used AMD GPUs before due to issues with their drivers under Linux.
Ampere Altra
They are now shipping. Some reviews have started to surface. Apparently the 80 core 3.3Ghz model is faster than an AMD Threadripper at compute performance. I made inquiries about getting one but never heard back from my server supplier. I will have to ask again.
12 December 2020
12th of December
Farm status
CPU only
Running Rosetta
Intel GPUs
Off
Nvidia GPUs
Ran some Rosetta. Currently off
Raspberry Pis
Pi3s running Einstein. Pi4s running Einstein and Rosetta.
For news on the Raspberry Pis see Marks Rpi Cluster
GPU issues
I went to use a couple of the Nvidia GPU machines and found that the GPU was missing in BOINC. This is caused by the "GPL Condom" code added to the 5.9 Linux kernel. It blocks drivers that use both GPL and proprietary symbols. All of my Nvidia GPU machines were on the 5.9.6 kernel.
The display part works fine, its the CUDA and OpenCL capabilities that are blocked, which means I can't use the GPUs for compute work. To work around this I ended up down-grading two of the machines to the 4.19 kernel that Debian buster is currently running. While its easy to select an earlier kernel on the Grub boot menu, its another thing to remove the newer kernel(s) from the machine without reinstalling and that took a bit of experimenting.
CPU only
Running Rosetta
Intel GPUs
Off
Nvidia GPUs
Ran some Rosetta. Currently off
Raspberry Pis
Pi3s running Einstein. Pi4s running Einstein and Rosetta.
For news on the Raspberry Pis see Marks Rpi Cluster
GPU issues
I went to use a couple of the Nvidia GPU machines and found that the GPU was missing in BOINC. This is caused by the "GPL Condom" code added to the 5.9 Linux kernel. It blocks drivers that use both GPL and proprietary symbols. All of my Nvidia GPU machines were on the 5.9.6 kernel.
The display part works fine, its the CUDA and OpenCL capabilities that are blocked, which means I can't use the GPUs for compute work. To work around this I ended up down-grading two of the machines to the 4.19 kernel that Debian buster is currently running. While its easy to select an earlier kernel on the Grub boot menu, its another thing to remove the newer kernel(s) from the machine without reinstalling and that took a bit of experimenting.
24 November 2020
Ryzen 5900X issues
Ryzen 5900X update
As I mentioned in my previous post I received one Ryzen 5900X CPU. I have replaced one of the Ryzen 3900X with it. Before swapping out the old CPU I did a BIOS update to get the latest AGESA (1.1.0.0) to support the Ryzen 5000 series. The Linux kernel doesn't like it. When it boots up it throws quite a few of these messages:
EDAC amd64: Error: F0 not found, device 0x1650 (broken BIOS?)
It still manages to start up though. I haven't tried doing any crunching with it.
From internet searches it would seem there are upstream patches for device 0x1650 to the EDAC but I'm not sure what kernel one needs to be running to get it. The machine is running a 5.8.10 kernel. I tried 5.9.6 from Debian bullseye but that didn't help. There is a 5.9.9 in testing which I haven't tried. If you run Linux check you can get a (very) recent kernel before you install your nice shiny new Ryzen.
Update 1 Dec 2020
I should point out that the machine still works despite the 24 EDAC errors above.
I have raised a bug with Debian but haven't had any response. I don't know which kernel will contain a fix. I tried the 5.9.9 kernel and that didn't help.
Debian have an upcoming release of bullseye, expected in March 2021. The 5.10 kernel has been listed as an LTS (Long Term Support) version so I wouldn't be surprised if they put the 5.10 kernel into bullseye and hopefully it contains the necessary patch.
Update 19 Dec 2020
Debian closed the bug. The patch is in the 5.10 rc7 kernel. While 5.10 has been released by the kernel team it had a couple of bugs with the RAID code so they released a 5.10.1 already. Debian experimental have this at the moment and I expect it will be pushed up into bullseye in time for its release.
Subscribe to:
Posts (Atom)