This project aims to create a four node Raspberry Pi cluster and report on the building model, file systems and models types after research. It will attempt to get a live, working, demonstrable model, that can be used and tested on in class. Slide show documentation of the steps and processes involved in creation and setup will be presented from various sources, but will also include personal modifications as necessary.
Basic knowledge in cluster computing will be presented; along with an explanation of the many primary uses, benefits, set-up, and utilization of such systems. With considerable mysteries and problems being solved and explored through clusters and super-computing, and with the advent of quantum computing being researched and developed, understanding the nature of how such systems congregate will provide standard users with tools and knowledge that can be advantageous.
* Raspberry Pi 3 Model
System-on-chip Broadcom BCM2837
CPU 1.2 GHz 64/32-bit quad-core ARM Cortex-A53
Memory 1 GB LPDDR2 RAM | 900 MHz
• Heat Sinks Eleduino 10PCS Aluminum Blue
• MicroSDHC SanDisk Ultra 16GB
• USB Micro, 1ft Neon Green USB 2.0 A Male to Micro B
• USB Charger Anker 40W 5-Port Family-Sized Desktop
• Ethernet Cat6 Patch Cord
• Paint TAMIYA PS-40 Translucent Pink Polycarbonate
• Switch TP-Link 5-Port Fast Ethernet (TL-SF1005D)
• Case GeauxRobot Model B 4-layer Dog Bone Stack Clear Case
Clusters & Computing
Clustering, refers to groups of similar things which interact with each other to accomplish varied tasks. In computing, the term takes on an analogous meaning that can refer to either clustering components, or groups of machines working together to perform similar operations and functions. With load-balancing capabilities through single software programs, managed by each node, or computer, they can provide answers to many of the most problematic challenges we face. Each node works independently to achieve modest tasks, which are also done in unison to accomplish greater goals. Typically referred to as parallel computing, parts of a problem are divided, while collectively carrying out solutions, providing scalability and increasing fault tolerance for certain environments.
Clusters can range from two to thousands of machines, referred to as farms. Ranges in size differentiate tasks performed, which include everything from complex scientific calculations and algorithm work, to web server duties that can be applied using them. With some of the biggest problems being solved through parallel computing, understanding the nature of how such systems congregate can give standard users extremely powerful tools to work with.
Fail-Over: Two or more connected computers with separate heartbeats that have connections between hosts. These heartbeats monitor processes to see if everything necessary is being used and provide fault tolerance when a failure happens, by having other machines come in and compensate for the loss.
Load Balancing: Checks to see which machines are least busy and sends requests to them.
HIGH Performance: Configured to give data centers better performance by spreading processes through numerous machines, while lessening the load of each.
Non-Uniform Memory Access Model: Uses shared access to memory whenever it executes code.
Distributed Shared Memory Model: Used with both hardware and software to provide abstraction layers for distributed memory.
The history of cluster computing is often referred to as one of the primary motivations for the development of networks. In 1967, Gene Amdahl of IBM, published an academic paper entitled: “Validity of the Single Processor Approach to Achieving Large-Scale Computing Capabilities” arguing that theoretical speedup in latency of fixed task workload could be expected when resources improved. This gave birth to Amdahl’s Law, which is often used in parallel computing.
Beowulf: Named for the epic poem of the hero of Geats and a specific computer built in 1994, Beowulf clusters are sets of identical computers in a LAN network, which share processing, libraries, and programs between them, without any specific software used to define the. A bramble is a Beowulf cluster made from Raspberry Pi’s.
Aiyara: A Beowulf specialization cluster built from commodity hardware, which is considered low power because it must be built with processing units that produce less heat. Based on a Thai word that means an elephant reflects under software stacks; the name originally referred to the first ARM-based cluster.
Hadoop File System: An Open Source framework for processing large amounts of data and storage, designed to be fault tolerant. Files are split into blocks which are then replicated into separate Data Nodes. HDFS uses a default block size of 64MB (can be adjusted).
The Hadoop framework is split into four modules, each with specific tasks. The Common module works with libraries and utilities for other modules, while the Distributed File System stores data on commodity machines. YARN (Yet Another Resource Negotiator) is a resource-management platform for clusters and MapReduce implements the programming model for large scale data processing on clusters.
Building My Bramble
My bramble was a modified version of a tutorial found on Makezine that I found inspirational. To make my bramble, I initially needed a USB charger to remove the need for individual power supplies for each Pi, and a 5-port miniaturized switch, each that took 5V respectively (the same voltage as the Raspberry Pi’s). I was able to find the USB charger on Walmart.com and the switch on Amazon for decent prices. From there, I went ahead and purchased a translucent polycarbonate spray paint from TAMIYA to ensure the coats I painted would bond to the GeauxRobot case properly. The blue heat sinks and neon green USB cables were just added personal expenses for aesthetical value. You don’t have to get fancy with any of the parts. Pictured below is an image of parts sitting on my desk.
The pictures below show the paint work on the GeauxRobot Dog Bone case. It was easy to do, it just required peeling the stickers and making sure I had a good surface area to work with. Even though the paint dried quickly, I let these sit overnight to avoid smudging or scratching them. I felt the neon pink was a nice touch to provide uniqueness to my project.
The images below show what the case looked like when it was finished. I used the rubber stickers that came with the GeauxRobot case and placed them underneath the switch, which gave me some wiggle room to attach it to the case with zip ties. I also used two zip ties to attach the USB charger and a Velcro strap to secure it better.
One all the Pi’s are seated into the case nicely and the attached charger and switch are in place, it’s time to install Raspbian Jessie (basically Debian) on each microSD card. Be sure to check the hashes on the download and to format each card pre-installation (HOW-TO). Pictured below are what the physical cluster looked like when everything was put together. I chose not to use Velcro on the cables, simply because of the size of the project.
Configuring and Troubleshooting
Once Raspbian was set on each Pi, I went ahead and made a few configuration changes to make things run a little bit smoothly. The first thing to note is that Debian defaults to a non-US keyboard and this can cause issues with characters like ~ and @. To make sure the keyboard preferences are set to en-US, you can modify settings in terminal using the sudo dpkg-reconfigure keyboard-configuration command. This will provide you with prompts on which keyboard you would prefer to use. You can also use usermod, useradd and sudo passwdcommands to make changes there as necessary as well. For the purposes of this paper, I did not.
After my initial profile configurations were set on each Pi, I in turn expanded the file systems on each card. The usual distribution images are about 2GB and a portion of the card tends to be unused when you copy images over to it. Expanding the file system here provides you with more space on the cards. To do this, I simply opened a terminal and used the sudo raspi-config command and chose the expand file system option.
The tutorial I was used suggested reconfiguring the /etc/hosts file, which I did on each Pi. I used a simple IP addressing scheme to keep things clean. To see the file, you can type cat /etc/hosts from terminal and to edit the file, sudo nano /etc/hosts using ctrl +x to save the file (use whichever text editor you prefer obviously).
::1 localhost ip6-localhost ip6-loopback
192.168.x.x rpi0 rpi0.local rpi0.lan
192.168.x.x rpi1 rpi1.local rpi1.lan
192.168.x.x rpi2 rpi2.local rpi2.lan
192.168.x.x rpi3 rpi3.local rpi3.lan
The next thing I went ahead and generated some SSH keys to enable each Pi password-less access, which is important later. For a more descriptive tutorial on the subject check here. For the quick overview, if you don’t already have any keys, you can type ssh-keygen -t rsa -C whatever@pi in ~/.ssh, use ls ~/.ssh to see if authorized_keys are displayed and cat ~/.ssh/id_rsa.pub to make sure your key is there. After, you can copy the key to your Pi and append it to your authorized_keys, over SSH by typing: cat ~/.ssh/id_rsa.pub | ssh name@yourIPaddress ‘cat >> .ssh/authorized_keys’ Lastly, you want to test everything by typing ssh name@IPusedtoTest
So, one of the initial mistakes I made, which I had to backtrack and troubleshoot was not making copies of the keys I created at this point. Run ssh-copy-id pi@IPofMachinesUsed to avoid any errors with Ansible later. Another thing you may want to knock out right now is the sshpass installation required for Ansible. This also caused a lot of errors and grief because I didn’t realize I hadn’t done this. Take a minute to sudo apt-get install sshpass to avoid problems with the next step.
Ansible is a great program that allows infrastructure to manage and deploy software via ssh and remote nodes. You can basically create what are known as playbooks that act as scripts with YAML which helps to automate various tasks you may want. Below I copied the pre-requisites and a few of the commands I played with. You can view the full tutorial here.
Jinja2 Stand-alone template engine for Python
PyYAML YAML parser/emitter for Python
parmiko Python SSHv2 chan lib
httplib2 HTTP client lib
sshpass Non-interactive ssh password authentication
Since Raspbian is Debian based, we can use sudo apt-add-repository ppa:ansible/ansible -y then sudo apt-get update && sudo apt-get install ansible -y and add ansible PPA to get this all set up. You are also going to want to edit the sudo nano /etc/ansible/hosts file with your Pi’s addressing schema.
Here you can see some of the SSH problems I mentioned above because I didn’t copy my keys, or setup sshpass pre-install. If you do run into this issue, you can still use the -k flag to prompt for passwords at the end of your ansible statements to test pings. You can also use the -vvvv flag at the end of ansible statements to troubleshoot any errors. After resolving both the key and sshpass issues, I was able to get a solid output without needing the -k flag.
Ansible has a lot of awesome options that allow you to run all kinds of YAML playbooks for automation. Here are some basic argument commands that can be used for general purposes and there is a great video tutorial on the subject that can be seen here. The tutorial also shows how to modify the etc/ansible/hosts file mentioned earlier.
Pinging nodes: ansible name -m ping (name should be one picked in host file)
Memory usage on all machines: ansible all -a “free -m”
Detailed settings: ansible all -m setup
Piping all IP’s: ansible all -m shell -a “ifconfig | grep inet” -s
You can also add the fork=number argument if you to specify how many nodes should handle what simultaneously.
In closing, I would use playbooks to direct my cluster in certain ways. I made a lot of mistakes during this project and it was a real learning experience for me. I had never even heard of ansible, so trying to learn and troubleshoot a completely new platform type for me ended up being somewhat challenging. Most of my parts took longer to ship and I had to order a few things I missed in the first order a second time. I lost the BlinkStick Nano that I was going to use to write a fun script for to add to the project (which I’m saddened by). I also lost the original screws the GeauxRobot case came with, but found replacements. Below however, are the results of all my work and I’m proud of the fact that I got to resolve a lot of the issues I came across with the minimal knowledge I had on the subject. Below is a screenshot of some of the above commands running successfully.