30 October 2016

30th of October

Farm Status
Intel GPUs
Running Seti and Seti beta overnight

Nvidia GPUs

Raspberry Pis
Running Einstein BRP4 work

Other news
I finished the memory upgrades for the Intel GPU machines. They now have 16Gb of memory.

I've ordered 3 of the WD 1Tb SSHD drives (Solid State Hybrid Drives). I will be putting two of them into the Nvidia machines to see how they go. If things work out I will order more to replace the drives in the Intel GPU machines.

The Pi Drives have been shipped from the WD store in America. I haven't heard from the freight forwarding service yet.

I have reimaged another two Pi's this weekend as they seem to be getting a lot of inconclusive results, although it might not be the Pi's fault. A lot of them seem to be paired with Intel GPUs which we know produces invalid results.

I'm looking at swapping the graphics cards around in the Nvidia machines. I have dual GTX750Ti's in the 6 core/12 thread machines and a single GTX970 in the i3 machines. The 6 core machines can only run a single slot at PCIe x16 and the second slot runs at x4. The i3 machines are able to run dual PCIe x16 slots.

22 October 2016

22nd of October

Farm status
Intel GPUs
Three running CPDN work. The rest are running Seti

Nvidia GPUs
Two running GPUgrid work

Raspberry Pis
All running Einstein work

Other news
For the Raspberry Pis I moved the root partition across from the MicroSD card onto an external hard disk for the one Pi's that I was using to test the process. See the blog post prior to this for the process I used.

After successfully moving the root drive on the Pi I decided to order a couple of PiDrives, enclosures and cables from the states. Its the first time I've tried using one of the freight-forwarding companies so it will be interesting to see if they actually turn up and how long they take. Its cost USD 98.80 so far. I have to pay the freight forwarder as well and on top of that there is the exchange rate. Its certainly not a cheap option but then PiDrives aren't available in Australia.

The memory upgrades for the Intel GPU machines arrived during the week. I have already upgraded 5 of them. The other 3 are running CPDN climate models so I will wait until the climate models finish before upgrading them.

The GTX1050 and GTX1050Ti were officially released however most suppliers have run out of stock. Looking at the EVGA ones specifically they don't have a VGA connector which is fairly normal these days but they don't recommend using an adaptor either. That's an issue for me as my KVM's only have VGA connections.

Moving a Rpi root partition

The Raspberry Pi uses a MicroSD card as its primary storage, Unfortunately it is also prone to corruption. The following is the process I've used to move the root partition from the SD card onto a USB storage device.

I first tried it on a Maxtor OneTouch II external hard disk that I had siting around. It gave me an error when I tried to write the partition table to the disk, so it may not work on all devices. I also tried it on a Sandisk Cruiser Switch (thumb drive) and lastly on a Seagate Expansion external hard disk. The last two worked fine. As they say your mileage might vary.

The Pi isn't very good at powering USB devices like hard disks so use a powered hard disk (ie one that comes with it own power pack) or you could use the PiDrive adaptor cable that supplies power to both the Pi and the external hard disk. Thumb drives should be fine as they require minimal power.

I can't take credit for the instructions, I got them from a Raspberry Pi forum post from 2013. It can be found at https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=44177
The instruction below are done in a terminal window.

Start-up Pi
Start off with a clean Jessie or Jessie Lite image on the micro-SD card. Boot up and change a few settings via Raspi-Config (sudo raspi-config). The HDD or thumb drive should not be plugged into the Pi at this point.

- Change_password
- Change locale as appropriate
- Change the timezone as appropriate
- Change memory split to 16Mb
- Exit and reboot

Find the device name
Plug in HDD or thumb drive and type “tail /var/log/messages” (without the quotes). The log should looks something like this:

Oct 19 20:43:53 pie39 kernel: [512215.189155] usb 1-1.2: SerialNumber: L2069VYG
Oct 19 20:43:53 pie39 kernel: [512215.190065] usb-storage 1-1.2:1.0: USB Mass Storage device detected
Oct 19 20:43:53 pie39 kernel: [512215.208124] scsi host0: usb-storage 1-1.2:1.0
Oct 19 20:43:54 pie39 kernel: [512216.272762] scsi 0:0:0:0: Direct-Access     Maxtor   OneTouch II      023g PQ: 0 ANSI: 4
Oct 19 20:43:54 pie39 kernel: [512216.282521] sd 0:0:0:0: Attached scsi generic sg0 type 0
Oct 19 20:43:54 pie39 kernel: [512216.328166] sd 0:0:0:0: [sda] 195813072 512-byte logical blocks: (100 GB/93.4 GiB)
Oct 19 20:43:54 pie39 kernel: [512216.383210] sd 0:0:0:0: [sda] Write Protect is off
Oct 19 20:43:54 pie39 kernel: [512216.438164] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 19 20:43:55 pie39 kernel: [512216.629457]  sda: sda1
Oct 19 20:43:55 pie39 kernel: [512216.849474] sd 0:0:0:0: [sda] Attached SCSI disk

We can see that device sda has been assigned. If you got another device name (eg sdb) then change the commands below to refer to /dev/sdb instead.

Partition disk
Start fdisk so we can see the partitions and create one as needed. Type “sudo fdisk /dev/sda”. The command p will display the partitions:

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p

Disk /dev/sda: 93.4 GiB, 100256292864 bytes, 195813072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0b2461bf

Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1        2048 195809279 195807232 93.4G  7 HPFS/NTFS/exFAT

Lets delete it (D command) and create a new primary partition (N command). We'll use the defaults for everything. Once done we’ll write it to disk (W command). If you need to exit without writing it to disk then use the Q command.

My Seagate Expansion had 4 partitions. They are shown as extra lines as above with /dev/sda2, /dev/sda3 and /dev/sda4 under the Device column. I had to repeat the delete command for each partition.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (2048-195813071, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-195813071, default 195813071):

Created a new partition 1 of type 'Linux' and of size 93.4 GiB

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.

If you get an error message after that ioctl line then you know your device is not compatible like my OneTouch II.

Format disk
We now need to format the partition in the ext4 format with the following command “sudo mke2fs -t ext4 -L OneTouchII /dev/sda1”. The word after the -L parameter is the volume label. You might want to name it something other than OneTouchII.

mke2fs 1.42.12 (29-Aug-2014)
/dev/sda1 contains a ext4 file system labelled 'OneTouchII'
        last mounted on Sat Oct  1 22:35:25 2016
Proceed anyway? (y,n) y
Creating filesystem with 24476378 4k blocks and 6119424 inodes
Filesystem UUID: d335ad29-9708-478a-80aa-9dabf0e3cdeb
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Lets mount the disk by typing “sudo mount /dev/sda1 /mnt”

Check we can see it by typing “df –h” and we should get something like this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  1.2G   13G   9% /
devtmpfs        483M     0  483M   0% /dev
tmpfs           487M     0  487M   0% /dev/shm
tmpfs           487M  6.5M  481M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           487M     0  487M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M   43M  33% /boot
/dev/sda1        92G   60M   88G   1% /mnt

Copy root across
Lets get rsync installed so we can copy all the files. Type “sudo apt-get install rsync". This should install the rsync package.

Copy root partition across by typing “sudo rsync -axv / /mnt”. This can take around 15 minutes.

Point root to the HDD
Backup the boot.txt file by typing “sudo cp /boot/cmdline.txt /boot/cmdline.orig”

Display it by typing "cat /boot/cmdline.txt". It should look like this:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

Edit the file (sudo nano /boot/cmdline.txt) and change the root= to our new partition and add a rootdelay of 5 seconds. After we're done it should look like this:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait rootdelay=5

Edit /mnt/etc/fstab (sudo nano /mnt/etc/fstab) and add the following:

/dev/sda1    /   ext4    defaults,noatime  0       1

And then comment out the memory card entry (put a # symbol at the front)

#/dev/mmcblk0p2  /               ext4    defaults,noatime       0       1

It should now look like:

proc            /proc           proc    defaults          0       0
/dev/sda1       /               ext4    defaults,noatime  0       1
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
#/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

Reboot (sudo reboot)

16 October 2016

16th of October

Farm status
Intel GPUs
Running Seti overnight and some Seti-beta on the iGPU

Nvidia GPUs
Did a burst of Seti on two of them during the week, otherwise off

Raspberry Pis
Running Einstein BRP4 work

Other news
Einstein have started a "Tuning run" of their new Gravity Wave search. From initial reports work units use up to 1.5Gb of memory each. The 4 core/8 thread i7 machines only have 8Gb. I have ordered memory upgrades for 8 of them.

Two of the Raspberry Pi'3's were throwing validation errors on the work they completed so I reimaged and reinstalled them. Fortunately it doesn't take too long to rebuild them. The Rpi is prone to having memory card corruptions from time to time. I've read that I doesn't happen if you use some other media for the root partition (eg an external HDD or USB memory stick).

I was running a test Seti multi-beam app as the developers are looking at why the Intel iGPU's throw so many errors. They did some code changes which made things worse. They are looking at trying to disable the optimisation that get applied by the OpenCL compiler.

09 October 2016

9th of October

Farm Status
Intel GPUs
Running Seti work overnight

Nvidia GPUs

Raspberry Pis
Running Einstein BRP4 work

Pi news
For the last few weeks I have been running the optimised Raspberry Pi app and having all sorts of issues. I have taken all 9 of the Pi3's back to Jessie (from Stretch) because tasks take around 1/3rd longer under Stretch.

The in-place app has issues, however the out-of-place app seems to work fine. The drawback with the out-of-place app is it needs more memory and so I am limited to running 3 tasks at a time to fit within the 1Gb of memory the Pi3 has.

Stock app approx 42k seconds per task x 4
In-place app approx. 23.7k seconds per tasks x 4
Out-of-place app approx. 18.8k seconds per task x 3

The author of the optimised app is looking into the issues with the in-place. Hopefully we'll get it working and it will be the app of choice.

In other Pi-related news our Raspberry Pi team quietly crossed the magic number of 3.14 million credits. I was hoping we might have got a mention on the official Raspberry Pi blog or Magpi (their magazine) but we haven't so far.

I am also trying to sketch together a design for a Pi rack as I am annoyed with the Pi's being scattered around one corner of my computer room. The idea is to get the drawings done in some CAD software that can then be used by a 3D printer to make most of the parts. More on this once I've made some progress. I do have rough hand-drawn ones but need to get it done properly.

Non-Pi news
GPUgrid is having issues with providing sufficient tasks to keep all the crunchers busy. They are also awaiting the release of the CUDA 8.5 toolkit. The CUDA 8.0 toolkit was released about a week ago so I am not sure how long it will take before 8.5 becomes publicly available.

CPDN had a bunch of new Weather at Home2 models but they seem to have all disappeared at the moment as is typical with them - Lots of tasks then none.

I'm still trying to get Seti credits up to the same level as Einstein and Asteroids (35 million and 34.5 million respectively). Unfortunately Seti doesn't give as much credit as the other two projects so it takes longer to get to the same level. Seti has been giving out a lot of noisy work units recently that finish with an overflow error in 3 seconds flat.

The weather is getting increasingly warmer and so its harder to run the farm 24/7. At the moment I am running work overnight only.