Hossein Ghorbanfekr

Data Scientist at VITO

1 minute read

In this post, I explain steps on how to setup a test but scalable high-performance computing (HPC) cluster using Raspberry Pi and with a focus on data science. The experiences learned from this tutorial are intended to hopefully help you to build your real HPC cluster, e.g. with hundreds of compute nodes, which support both interactive and command-line interface.

Feature implemented

  • Slurm workload manager
  • Batch job submission
  • Jupyterhub service integrated to Slurm
  • Network file share (NFS)
  • User and group disk quota
  • Conda package management for Python/R
  • Environment module management using Lmod
  • Support parallel MPI applications integrated with Slurm

Note: Based on my experience, setting up a real HPC cluster is very similar to what we are doing here using Raspberry Pi.

Info: Generic resource scheduling such as Graphics Processing Units (GPUs) and Intel Many Integrated Core (MIC) processors are supported by Slurm and they can be easily added later through a flexible plugin mechanism on real HPC systems.

For a full description and required configuration files please see https://github.com/hghcomphys/raspi-hpc-cluster

You May Also Enjoy

Image Outlier Detection: Mahalanobis Distance

23 minute read

In this post, I will discuss how Mahalanobis distance (MD) can be used to detect image outliers from MNIST handwriting digits dataset as an example. Outlier...

Create a GSM Remote Alarm System

6 minute read

GSM Remote Alarm System is an experimental implementation of a broader idea of how one can build a home alarm system. This post shows my attempts to remotely...

How to Build a ESP32 Weather Monitor Module

6 minute read

Here I will show my attempt to build a wireless weather monitor module, using ESP32 microcontroller, that connects to the internet via WiFi and pulling some ...