22nd of December

Farm status
CPU only
Idle. Have been running early mornings.

Nvidia GPUs
Off

Raspberry Pis
All running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
I have been running the Ryzen 5900X machines early in the morning for the Universe@home project. They are an astronomy project studying black holes at the moment. The 5900X can do their work units between 30 minutes and 1 hour when running 24 at a time.

I had another look at the cost of upgrading the Ryzen 5900X to Zen 4 and its still rather expensive. In addition there doesn't seem to be any 32GB DDR5 memory modules available that can do 6GHz which is the sweet-spot for Zen 4. There are a few 16GB modules but they don't recommend using 4 memory modules due to timing issues. Another concern is the power consumption that a Zen 4 CPU will use and the additional cooling needed.

13th of November

Farm status
CPU only
Off.

Nvidia GPUs
Off.

Raspberry Pis
Running overnight.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
As you can see all the big crunchers are off. Its summer in Sydney so it can get hot. While I have air conditioning I won't use it to cool the farm, I would rather turn the farm off. Not only is this better for the environment but it also saves my electricity bill.


Intel ARC
I had a quick look at the Intel ARC A770 but decided not to get one. The Linux support for them hasn't landed in Debian yet and performance is about the same as the RTX 3060 Ti, which I already use. To do a firmware update one has to use the Intel Management Engine, which of course is only available on Intel based motherboards. Lastly BOINC and the BOINC-based projects don't currently support them. This should improve over time but I think its too early to get one.

9th of October

Farm status
CPU only
Running Universe@home part time.

Nvidia GPUs
Off

Raspberry Pis
Running Einstein@home BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Ryzen 7000 availability
I priced an upgrade for my Ryzen 5900X machines. I would need a new CPU (Ryzen 7900X) at $949, Noctua NH-U12A cooler at $199, ASUS X670-P motherboard at $499 and lastly 64GB of DDR5-5600 memory at $599. Giving a total of $2,246 in AUD.

I would aim for faster memory as the sweet spot for the Ryzen 7900X is 6GHz and that would cost more of course, assuming I could even find it. Oh and then there are NVMe drives.

As I see it we aren't quite there yet. DDR5 memory is too expensive, not much faster than DDR4 and in too small capacities. You can't even get PCIe gen 5 NVMe drives at the moment. Its going to take a while for the 6.0 kernel to get into a stable Linux release giving the necessary driver support. I think I will have to wait a bit longer.

22nd of September

Farm status
CPU only
Off

Nvidia GPUs
Off

Raspberry Pis
All running einstein BRP4 work

For news on the Raspberry Pis see Marks Rpi Cluster


Bill shock
I've been limiting the larger crunchers from use to keep the electricity bill under control. At the moment its been the Pi's running 24/7 with bursts of activity from the x64 machines on weekends.


RTX 4090 announced
Nvidia announced their Ada Lovelace architecture high-end graphics cards (RTX 4090 and 4080). They can use up to 600 watts, so I won't be looking at them. I currently have a number of RTX 3060 Ti cards but if Nvidia keep increasing power consumption I will move to their lower powered models. They haven't announced the rest of the range so I don't expect to be upgrading any time soon, besides the existing Ampere architecture cards are now being discounted quite heaviliy.


Other news
I'm still waiting for AMD Zen 4 to be available at the retailers so I can work out how much its going to cost to upgrade the Ryzen 5900X machines. Given the price of DDR5 memory at the moment I may wait a while for prices to drop.

23rd of August

Farm status
CPU only
Off. Had the Altra running on the weekends.

Nvidia GPUs
Have been doing Einstein and Milkyway on and off.

Raspberry Pis
All running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
There hasn't been any work from Rosetta@home so the Ryzen 5900X's have been off. I had the Altra running for a couple of hours on weekends doing Einstein BRP4 work.

The Raspberry Pis have doubled with an additional BitScope Edge Cluster 12 being added.

31st of July

Farm status
CPU only
Off. I had the Altra running Einstein BRP4 work earlier today.

Nvidia GPUs
Three running Einstein FGRP5 work.

Raspberry Pis
All running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
As you would gather from my other blog the Rpi's have moved into an EC12 which had me occupied doing the assembly and some reconfiguration of support nodes.

The Altra had a bit of a run so it did a bunch of Einstein BRP4 work units. The x64 machines didn't get to run as Rosetta doesn't appear to have any work available unless I want to run Virtual Box work units.

There is talk of the newer Nvidia cards using up to 800 watts, so I will probably be skipping them. You would think Nvidia would be reducing power consumption as they switch to smaller manufacturing nodes, but they appear to be getting worse.

9th of July

Farm status
CPU only
Off. Altra has been running today.

Nvidia GPUs
All running Einstein and Milkyway work.

Raspberry Pis
Pi4's running Einstein BRP4 work.


Other news
I had the Altra running for a few hours today. Its been chipping away at the Einstein BRP4 work. I hope to have it going again tomorrow.

I am waiting on more information of AMD 7000 series CPU's. I expect to upgrade the Ryzen 5900X machines to 7900X machines, but it looks like that will mean new CPU, Motherboard and Memory. Rumour has it they'll be announced in September.

Debian have just done a 11.4 point release, so a few updates to apply to the farm. Most of it seems to be the Nvidia drivers along with a kernel update.

26th of June

Farm status
CPU only
Off

Nvidia GPUs
Had two running Einstein FGRP5 work overnight

Raspberry Pis
Pi4's running Einstein BRP4 work. Pi3's running Einstein BRP4 over the weekend


Other news
I had the Altra going for a couple of hours so it did 156 work units in that time. It is rather noisy so I don't run it for long periods.

No work from Rosetta@home (unless I want to run their Python Project - Which I don't) so the Ryzen 5900X machines stayed off.

12th of June

Farm status
CPU only
Off

Nvidia GPUs
Running Einstein FGRP5 work overnight

Raspberry Pis
Running Einstein BRP4 work

For news on the Raspberry Pis see Marks Rpi Cluster


Switch upgrade
I got a couple of new 2.5GbE switches. They are Dlink X106XT switches. They have a single 10GbE port and 5 x 2.5GbE ports. They are an un-managed switch but one can prioritize a particular 2.5G port via a switch on the back. It has an annoying LED strip on the front. Unfortunately one can't turn it off but can have it display a white colour instead of the RGB. The 10GbE port is intended to be used as an uplink port so you can integrate them within a 10G network and get the best performance.

I had a QNAP 5 port 2.5GbE switch but the Dlink should be able to push the network speed a bit more. I've removed the QNAP switch. My compute nodes all have 2.5GbE network cards installed.


Other news
I had the Altra running for an hour or so today. It only takes an hour to do Einstein BRP4 work units, so I let it run for a bit and churn out 78 work units. It has an Ampere Altra 80 core (ARMv8) CPU running at 3GHz. I keep two cores available for other tasks.

Rosetta@home is out of work at the moment so no point running the Ryzen 5900X machines.

29th of May

Farm status
CPU only
Off. Have been doing some Rosetta work.

Nvidia GPUs
Running Einstein and Milkyway work.

Raspberry Pi's
Running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Zen 4 news
The information about Zen 4 is sounding quite promising with motherboard makers and AMD making some announcements at Computex 2022. I will look at upgrading the Ryzen 5900X machines to the equivalent Zen 4. We're still waiting for more details from AMD about CPU specs. Interestingly AMD have said the CPU's will have integrated graphics as well as supporting PCIe 5 and DDR5 memory.

Initially there won't be anything that uses PCIe 5, apart from maybe some M.2 SSD's but that will change over time. The CPU socket has changed to AM5 so that will require a new motherboard and newer DDR5 memory. There is also talk of M.2 SSD's getting bigger, probably to the 22110 size (110mm length) to support PCIe5.


Energy pricing
We have been warned to expect electricity prices to increase up to 140% this year as the wholesale electricity prices have jumped. Energy efficiency is certainly on my mind these days. They are also expecting natural gas prices to increase although not as much. Unfortunately my home doesn't have a north facing roof so adding solar panels wouldn't help much, not to mention I have a fairly small roof area so couldn't fit many panels.

8th of May

Farm status
CPU only
Off

Nvidia GPUs
Two running Einstein and Milkyway work.

Raspberry Pis
Running Einstein BRP4 work.

For news on the Raspberry Pis see Marks Rpi Cluster


Work mix
Rosetta ran out of work so nothing for the CPU's to do. Einstein continues to have limited work for GPU's. GPUgrid doesn't seem to have any work. Milkyway seem to have sorted out their server issues and is providing GPU work when requested, not withstanding their bug where if you report and request work at the same time it doesn't give any work.

24th of April

Farm status
CPU only
Had the Ryzen 5900X machines doing Rosetta@home for a day.

Nvidia GPUs
Had all machines doing Einstein@home and Milkyway@home.

Raspberry Pis
All running Einstein@home.


Work mix
The weather allowed for running Rosetta@home on the Ryzen 5900X machines so I did a burst of work for them. I had a couple of the Nvidia GPU machines doing Rosetta on their CPUs.

I also had the Nvidia GPUs doing Einstein@home and Milkyway@home work on the RTX 3060 Ti's. Sometimes Einstein doesn't give GPU work when requested. I assume this is caused by their running out of signal candidates for the FGRP5 work and having lots of GPU equipped hosts asking for work.

10th of April

Farm status
CPU only
Off

Nvidia GPUs
Off

Raspberry Pis
All running Einstein@home BRP4 work

For news on the Raspberry Pis see Marks Rpi Cluster


Work availability
We're still running from time to time, depending on the weather. In a break from the rain its sunny and hot with more rain forecast for later in the week. The Pis have been running constantly. I had the Altra running for a couple of hours yesterday doing Einstein BRP4 work.

The other machines have generally been off because they are excluded (by the project) from doing BRP4 work. I usually use the Nvidia GPU machines to run the Einstein FGRP5 search but there isn't much work available for them or from GPUgrid.

Milkyway which I also run on the Nvidia GPUs has been having server issues where it says it doesn't have any work but the server status page shows there to be plenty.


Storage server update
I got the parts mentioned in my previous blog post and assembled it. I had to buy an over-priced 2nd hand GT 710 graphics card off eBay so I could have a display. The GT 710's have been replaced by the GT 730, hence the eBay purchase.

I assembled the new motherboard and swapped out the existing one. Everything else got reused in the new build. I didn't even install the operating system I just put the old M2 SSD into the new motherboard and off it went. The only thing I had to install was the Nvidia drivers.

I couldn't get the 32GB memory sticks to work so I used the memory from the old machine (4x16GB sticks). It seems none of the machines I have recognize the 32GB sticks.

20th of March

Farm status
CPU only
Off

Nvidia GPUs
Two doing Einstein, weather permitting

Raspberry Pis
Nine Pi4's running Einstein.

For news on the Raspberry Pis see Marks Rpi Cluster


Storage server update
In my last post I mentioned I was looking at updating the storage servers. I gave up looking for a Xeon E-2300 and instead have ordered an ASUS X570-Pro motherboard, Ryzen 5600X CPU and a Noctua cooler. I'll move PCIe cards from the old machine and I have memory. The Xeon E-2300 series and the Ryzen both have a 128GB memory limit, the difference is I can get the Ryzen and its cheaper.

I went with the Pro motherboard as it gives an extra M.2 slot and an extra PCIe x16 slot over the X570-P motherboard. I need to add a graphics card to the machine which only leaves 2 PCIe slots left for the SAS controller and 10GbE network card.

I tested 32GB ECC memory sticks in an X570-P machine (with a Ryzen 3600) and it wouldn't post. I changed to 16GB sticks and they worked. There is an option buried in the BIOS to enable ECC mode.

Given the Ryzen 3600 didn't recognise the 32GB memory sticks I decided to get a Ryzen 5600X in the hope it will. Its also a little faster than a Ryzen 3600 while still being a 65 watt part. Both the X570-P and X570-Pro motherboards have 4 memory slots. I have to use 4 x 32GB sticks to get to 128GB of memory. To get ECC memory support I have to use an X series CPU. The ones with graphics (the G series) don't support ECC memory, hence the need for a graphics card.

26th of February

Farm status
CPU only
Off

Nvidia GPUs
Off

Raspberry Pis
Off


Other news
Its not quite as bad as it seems. I have been doing bursts of work between the hot weather. It has also been raining heavily for the last week. The Nvidia GPU machines have been doing Einstein and Milkyway work when they can. I even managed to fire up the Ampere Altra for a couple of hours today (doing Einstein BRP4 work). The main limitation with the Altra is the noise it makes.

Last week saw Einstein@home run out of BRP4 work so all the Pis ended up with the Gamma Ray work which takes around 27 hours a work unit on a Pi4. I haven't tried running these on the Altra as I cannot commit to have the machine running for 14 hours which is my estimate on how long they'll take. They could be quicker as the Altra has faster memory (DDR4 @ 3.2Ghz) and more memory channels than a Pi4.


Storage servers
In unrelated news I am looking for replacement motherboards/CPUs for the storage servers. The existing ones work but I am hoping for something more modern. One of them currently has an i3-8100 (4c/4t) and is limited to 64GB of memory.

The Xeon E-2578 with an ASUS P12R-E motherboard and 128GB of DDR4 @ 3.2GHz memory looked promising but there doesn't seem to be stock anywhere.

12th of February

Farm status
CPU only
Running Rosetta work

Nvidia GPUs
Two running Rosetta (CPU) work

Raspberry Pis
Pi3's idle. Pi4's running Rosetta work


Playing catch-up
As mentioned in my last post I had the Nvidia GPU machines running, when the weather permitted. There were a couple of hot days where everything was off and it got to 32 degrees C. Rain and cooler weather returned and I had them trying to get their Einstein credits to similar values.

The two oldest RTX 3060 Ti machines are on 10.9M credits each so the newer ones were running non-stop trying to catch up. The newest had got to 10.7M credits when I last checked today and the other reached 10.9M credits.


Rosetta on all cores
Well not quite all cores. Rosetta suddenly has lots of work so I have most of the machines running it at the moment. The Ryzen 5900X machines have 48 tasks (24 each) going and two of the Nvidia GPU machines have 24 tasks (12 each). They usually take 8 hours to run.

5th of February

Farm status
CPU only
Off.

Nvidia GPUs
Three idle, one running Einstein.

Raspberry Pis
All running Einstein.

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
I finally got time to install the 4th RTX 3060 Ti and its running now.

The weather up until last week has been hot, so much so that all the machines and even the Chia storage server were off. It got to 33 degrees C in the room where the computers are! About 3 days ago we got cooler weather that allowed me to start everything up again and even get the GPUs running. Unfortunately it looks like we're heading back into hot and humid weather in about 2 days time, so the farm will probably be off again.

19th of January

Farm status
CPU only
Off

Nvidia GPUs
Three running Einstein and Milkyway (GPU) work

Raspberry Pis
Pi3's running Einstein. Pi4's running Einstein and Rosetta

For news on the Raspberry Pis see Marks Rpi Cluster


Other news
Its been hot and humid. The loft where the machines are has been in the 28-31 degrees (C) range for the past 3 weeks. Yesterday we had a break from the heat and things cooled off a bit so I could get some of the machines going.

It was so hot one of the hard disk drives in the external disk enclosure stopped working. When I went to remove it I couldn't hold it because it was too hot. I left the enclosure off for a couple of days. Its now back on-line. It has an 80 or 90mm fan in the back of the enclosure but it doesn't seem to move much air.

I still have one GTX 1660 Ti to replace. The three GPU machines running at the moment have all been upgraded to RTX 3060 Ti and 850 watt power supplies.


Future plans
I am waiting to see how the PCIe gen5 and DDR5 transition pans out. DDR5 memory is particularly expensive and in short supply at the moment. I'll probably replace the Ryzen 5900X machines with whatever comes next from AMD. The Intel chips are too power hungry at the moment to consider, but time will tell. Intel are on a 10nm process where the next gen AMD chips are looking at 3nm so should be more power efficient.

PCIe gen5 doesn't help with the GPU's as the PCIe bus isn't a bottleneck. Switching to PCIe gen5 SSD's doesn't make much difference either, sure I/O is quicker but the compute nodes don't do that much I/O. DDR5 memory on the other hand could benefit the number crunching, particularly on the CPU-only machines.