Loading 3D model… large boards can take a moment.
This project is a redesign of pi-cluster-one.
The planned scale of this design is 16 nodes, with a stretch goal of 64.
Its primary intended use is as a vehicle for learning about HPC cluster internals, hardware design, and the open source ecosystem for hardware design.
The building blocks that comprise this cluster are:
In addition to a regular gigabit ethernet network, the design of this cluster supports:
The left-right torus dimension is hardwired on the backplane, in two groups of four nodes. Remaining torus and tree network links require 2-pair UTP cables connected to JST connectors on the backplane.
| compute nodes | backplanes | supervisors | tree cables | torus cables | notes |
|---|---|---|---|---|---|
| 4 | 1 | 1 | 3 | 0 | 1D torus |
| 16 | 2 | 1 | 15 | 16 | 2D torus (4x4) |
| 64 | 8 | 4 | 63 | 128 |
| 3D torus (4x4x4) |
A compute board carries a Raspberry Pi 4B with support circuitry including a service processor for remote management and monitoring over CAN bus, a 5x7 LED bargraph for status display, a CAN adapter for the Pi, and a mezzanine slot for a communications adapter.
10 version 2 compute board PCBs were were received and are being hand assembled/debugged.
A CM4 version is planned.
The backplane supports power distribution, geographic slot addressing, a CAN mangement/monitoring bus, a wired-OR hardware barrier network, a tree network for optimized MPI collectives, and a 2D torus mesh network.
Five version 1 backplanes have been ordered, with PCBA assembly.
The version 1 supervisor design is being finalized.
Not yet designed, but availability of a PCIe lane on the CM4 makes this somewhat more interesting than before.
STM32 firmware on the compute board version 1 and 2 is fully functional. The firmware "personality" for the supervisor board is not yet complete (it will use the same firmware image).
Tools on linux for snooping the CAN bus with message decoding, pinging over CAN, reading slot address over I2C, etc. are functional.
Integration with powerman and conman for remote power and console management of a single crate is functional.
The basics of the CAN protocol are documented.
SPDX-License-Identifier: GPL-3.0-or-later
SPDX-License-Identifier: CERN-OHL-1.2
This project includes parts of:
The CAN control bus and LED matrix were influenced by similar features of the Meiko CS/2, a ground breaking supercomputer of the 1990's.
The communcations design is patterned after Blue Gene/L, circa 2005.
No comments yet. Be the first to ask about this board.