This is what the build looks like at a glance, with all of its components present as I wait for the remaining cables to trickle in from schmeBay. It is built into this ‘hot plate PC case’ from Antec that I must say has been an absolute joy working with, which surprised me considering its unorthodox design.

Now that all the parts are in, I can say absolutely none of those front ports you see will be functional in any way. That said, the power and reset switches will both be fully functional, and their LEDs will work swimmingly too. And yes, that is a 5¼″ and 3½″ combo floppy drive. It might be my favourite part of the build.

The plan is to run four operating systems on this machine (the maximum possible with MBR bootloading):

1. Windows 2000 Professional with the unofficial patched kernel

2. Windows NT 3.1 for universal compilation targeting of 32-bit NT

3. Windows 95 for universal compilation targeting of 32-bit DOS-based Windows

4. MS-DOS 6.22 with Windows 3.11 for targeting 16-bit real mode and Win16 applications, respectively

I am also considering swapping out MS-DOS + Windows 3.11 for Debian 7 so I can run somewhat recent Linux on it as well, since I read somewhere that’s the last Debian to support legacy Nvidia drivers for cards like my FX 500. I have another (much more archaic) i486 build that is supposed to run that combination anyway and will be taking NT 3.51 and NT 4.0 as well (absent here).

The boot drive holding all of these will be an 8GB UDMA 7 capable CF card (in UDMA 6) running on the primary master IDE, while the very nice Plextor DVD writer will be on the secondary master IDE.

These are the build’s mass storage drives: six (6) count 80GB Maxtor IDE hard disk drives that were open-box when I bought them as a lot on schmeBay. They present an opportunity for me to experiment with designing my own software JBOD and eventually software RAID solutions – something that fascinates me as software RAID is the norm today but was not at all twenty years ago when these parts were contemporary.

This is the supplementary IDE card (also from Maxtor 💛) which takes four of the six hard disk drives. The other two will occupy the primary slave and secondary slave positions on the motherboard’s IDE cables that run to the boot drive and the DVD writer.

This is my primary graphics card: an Nvidia Quadro FX 500. It sports 128MiB of DDR SDRAM for VRAM and a core that is fully certified for OpenGL 1.5 and partially certified for OpenGL 2.1. I bought it as a baseline for immediate-mode-style GL programming that is more era-correct than the other, more anachronistic GL 1.x chips I have (namely the GMA 3150 on Jericho).

This is a USB 2.0 PCI card. It provides the fastest data transfer ports that are realistically ever going to function on this computer, and unfortunately lacks any USB 2.0 panel headers (I looked everywhere and couldn’t find any PCI cards that did). The motherboard has two USB ports, but they are almost certainly 1.1 at best (slooooow!).

This is my system’s sound card. I bought it because I correctly anticipated my PCI port complement would be completely filled and didn’t want to risk slowing down the other high speed cards with something low bandwidth like a sound card. Like with the USB card, no modern case headers of any kind can be found on this, but at least it provides the usual complement of ⅛″ jacks on the back.

This fills in my other free ISA slot: an 8-bit ISA card (ergo, a PC card) that does CGA graphics. Stuck on the 9-pin port is a passive DB-9 to DB-15 connector so it can pipe to VGA monitors. This can be helpful when testing out Sirius DOS’s Subprime ABI for example, since it only uses text modes.

Finally, the pièce de résistance of the build: the Oxygen 402 from 3Dlabs.

3Dlabs was a trailblazer in the nascent industry of 3D graphics: in a world of SGI workstations and so-called Windows accelerators, they were the first to make CAD workflows practical on IBM-PC compatibles with the Oxygen series video cards, of which this 402 was their highest end model. They created the GLINT 300SX, which was the industry’s first single chip, 3D-capable graphics device that went to market. They also were a huge influence on OpenGL 2.0 as a member of its Architecture Review Board, and made huge strides filling in the budget end of the 3D graphics market with their Permedia brand. In short, 3Dlabs walked so that 3dfx, ATi and Nvidia could run.

Why do I have this card? Well, for the same reason I have everything else here: programming research. This card does original OpenGL in a way that is unlike any other card of its era, sporting four processing chips in parallel with a monstrous 32MiB of VRAM in total. I want my software to get stable framerates on this—within its feature capabilities of course—just as it should on the FX 500, Jericho’s Intel GMA, and any modern graphics chip you could think of.

This is what the cards look like installed. The Oxygen 402 is absolutely massive and barely fits in the case (though it does fit).

To accommodate the CPU’s power draw on my modern PSU’s 5V rail, I have acquired a PCIe-to-Molex breakout board that properly provides the 5V as well as 12V, converted on-board. That means all of the hard drives, the DVD writer and the combo floppy drive will draw from the PSU’s 12V rail, virtually eliminating the risk of overloading its 5V rail.

I’m excited to run this build, but I have to wait for the rounded IDE cables (which are thankfully UDMA capable), the rounded floppy cable and its card edge adapter, and some Molex adapters for those Shenzhen breakout boards to power it all. That big fan up top takes Molex too, which means there is not a single SATA power cable in sight. Very slick stuff.

All of these photos were taken with a Sony α7R IV camera with automatic exposure and manual focus and touched up in Adobe Photoshop CS2’s Camera Raw on Windows XP x64 Edition.

If you like this sort of content, follow along!

Subscribe now

Henen-nesw was always meant to be the unorthodox Linux dream build of mine since I first got into PC building back in high school. In those days, the Haswell refresh was the bee’s knees, and every gamer wanted to run a GTX 980 with a Core i7-4790K. 16GiB of RAM was considered overkill for most games and the coolest way to flaunt your wallet was by putting dual Intel 730 series SATA SSDs in RAID 0.

When Broadwell came to the desktop, people balked at its beefy iGPU and lack of complete SKU spread and promptly wrote it off so they could wait for Skylake. But I had a different opinion, and this meme explains why:

The only good driver (pun intended) worth a damn was in a golf cart, and that meant that getting the Iris Pro GPU in the Broadwell Core i5-5665C and Core i7-5775C was upgrading him to an ATV. To me this was—and still is—the epitome of cool.

Later on, I caught the mITX bug, and this was explored in the previous rendition of Henen-nesw that ran a Skylake Core i3-6100:

As I became more entrenched in low-level programming in my late teens and early twenties, things like the Xeon Phi began to fascinate me from a programming standpoint. As this weblog by now surely reflects, the programmability of a piece of computing hardware has come to dominate my tastes in purchasing and building computers.

I started Grovercomp with the intention of creating a poor man’s petascale supercomputer in my house. The cards for it have all been acquired and they sit in storage together with a handful of rackmount servers to await me having the funding to build them. But on the other hand, having more immediate access to a card for initial prototyping also seemed valuable to me; hence, I filled in that one PCIe slot normally provided for discrete GPUs with an awesome coprocessor from the same era of SATA SSDs and Intel market dominance.

One of the big gotchas about using Xeon Phi cards in current year is that mainline Linux has dropped their drivers from the tree entirely. One can cope with this by either using old distributions (10 is the last Debian to support it, for example), or by using these aftermarket drivers from the University of Peru built for RHEL-based distributions. I took the latter route in the interest of continued security updates, and while I eventually got it to work for a bit, the drivers proved unstable and the card would start randomly dropping and forcing a whole system restart to arrest the fans that rev up to max when the card disconnects. So, for now, I must run Henen-nesw without the card plugged in.

As a consolation for the readers, here is a gallery of the painting process of this 3120A which follows the mock-ups I made years prior:

The finished product:

Subscribe now

Soon, you will not find the Grovercomp weblog on log.grovercomp.ai, but rather grovercomp.xion.mt.

I found a new server box to house the first of what will eventually be a total of six nodes, each sporting three (3) Xeon Phi 5110P cards. It’s a rackmount chassis with proper cooling and came with its power supplies and connecting cables included. While it does have room for four (4) cards, the PSUs are not so permitting, plus it would save space for a high-bandwidth NIC if that is something we end up wanting to use.

I also finally got my hands on a 3120A for my personal home desktop, Henen-nesw. When I get its cover spray-painted, I’ll be sure to post it up on here.

Life is busy and full of adversity. I am honestly lucky to have found as much time and energy as I have for all of my pursuits in computing, considering the circumstances of the last few years.

Once built, I will leverage Grovercomp in all of its might as a target for the Oración assembler. I hope to run x86-64 code as natively on these coprocessors as I do on all of these CPUs, and have that integration reflect in the menagerie of tooling Xion will soon provide. Proofs of concept run on Grovercomp will ultimately set the stage for Anodyne. This was always the end game for this project.

Thank you for reading, and I’ll see you as soon as possible. Please consider subscribing if you haven’t already! It helps us out a lot.

Subscribe now

Chip simulation

Designing the Manitowoc chips that will be inside Anodyne and verifying properties about performance, efficiency, and programmability will require a significant amount of computing power and memory. Ideally, we want to simulate the processor close to or at the gate level, as this should allow a nice compromise between high simulation precision and usability on our end.

Something akin to vectorized bit matrices should be sufficient to emulate digital circuits fairly efficiently, and the dual 512-bit vector engines used in the Xeon Phis should be very effective for this. This is one reason why 1-bit precision matters a lot for Grovercomp; each bit-operation in sim can correspond to a single logic gate in the chip.

Simulating a small number of cores on a single Xeon Phi shouldn’t be particularly hard. Simulating the full Manitowoc chip with its thousands of cores may still technically be doable with a single Phi, though it gets much faster when the full cluster is in use.

Performance is a serious consideration; if we want to write software to test out the architecture and make tweaks to improve efficiency and programmability, we’d ideally like to get as close as possible to real-time. In practice, my current estimates are that Grovercomp will be able to simulate the full Manitowoc chip at around 30-100 kHz, mostly bottlenecked by network latency. Depending on NoC congestion, we may be able to get somewhat higher than that in practice, and if we really need high performance we can probably scale back to smaller numbers of cores.

One very valuable use of this will be in exploring advanced programming techniques for large tiled architectures. There is an enormous amount of efficiency that can be gained from intelligently laying out memory and computing power in space, turning optimization into a problem at the intersection of dynamic code execution and geometry.

SAT solving

Satisfiability (SAT) problems are very valuable for helping solve a wide range of verification problems, as they are a general-purpose tool for property checking across all possible states of a finite system. Verification is extremely important in chip design (it’s very hard to fix a bug when it’s baked into silicon), and so even exponential-time algorithms are on the table.

Luckily, SAT solvers tend to be much more efficient in practice than in theory. The precise nature of this efficiency gap between theory and practice is a major unsolved problem in complexity theory – closely related to the P vs NP problem. This makes it hard to reason in the abstract about how quantitatively beneficial different approaches to the problem may be.

One of the disadvantages of SAT solving on a big, highly-parallel compute cluster is that SAT solvers are not very well-suited for parallel programming. Yes, technically you could brute-force SAT problems in parallel to an enormous degree, but in practice you tend to get quasi-exponential speedups from more sequential code. This is mostly because SAT problems are bottlenecked by information learned about the problem, which tends to be of higher quality when built off previously learned information, and that is generated and tweaked so often that parallel SAT algorithms get bottlenecked by communication.

That said, there are many known ways that you can throw parallelism at preprocessing to speed up the sequential part. I also happen to be building a SAT solver that makes heavy use of vectorized bitwise operations to rapidly brute-force local searches in the SAT problem, which should hopefully provide more speedups while conforming very well to the capabilities of the Xeon Phis.

At the very least, it’s likely we’ll have many small SAT problems that need to be run as well, not just big ones, and so having 1200 cores or so will be beneficial regardless.

Superoptimization

The general approach I am taking toward developing the Manitowoc chips is to start out with a well-mapped landscape of the tradeoffs of different chip designs and computing techniques, combined with some statistics on code structure, to use as a foundation. Once the foundation is laid and a starting chip is available, then an evolutionary process can be used to hill-climb to a more ideal architecture that is capable of running ordinary code without enormous programmer effort while simultaneously doing so as efficiently as possible.

When you have a clean-slate chip design, you have the luxury of being able to scrap all the parts of legacy instruction sets that waste transistors while no longer being used enough to justify their existence from an efficiency perspective. In fact, anything can be on the table if you’re ambitious enough.

The big problem with this is that it means the instruction will may be changing constantly during development of the chip, which can make the simulated chip a very difficult compilation target!

The solution is superoptimization, which is simply just search-based code optimization. You take the space of possible instruction sequences, treat it as a search space, and ask what the fastest possible sequence of instructions are that do the same thing as the original code. This produces near-optimal code with little more than a depth-first search algorithm and a lot of time. New instruction sets could likely be generated by the same code that generates the CPU simulator.

This is of course an exceptionally hard problem in theory as you don’t just have the exponential search problem of finding the optimal code, but also the exponential problem of verifying that the code is correct across all possible inputs.

In practice though, it’s not completely impractical as there are many ways of optimizing the search (and of course is a lot easier if you have an entire compute cluster available rather than just a laptop). The existing academia demonstrates that this is a pretty promising approach in terms of the performance of the code it generates, but I’m far less impressed with the search algorithms they use to find such code. Throwing Grovercomp at analyzing codespace for shortcuts would be well within the realm of possibility, as would being able to optimize huge amounts of code quickly.

Thanks for reading! If you enjoy my work, please support me with a subscription to Bzogramming, my main newsletter. One-time donations can be made to Alexander via PayPal to fund the build. Until next time!

Subscribe now

Since we met, there has been a galactic collision of our lives’ work. Right now this is a nebulous soup of stardust, but in time it will reform and accumulate into what we call Project Anodyne.

Our main task as the masters of this new universe is to accelerate its accretion to within our lifetimes. We don’t want to have done every detail of this without collation, ageing out and leaving the corpus to be reconstructed by future minds, if it isn’t lost entirely. (Yes, that is a lot more common than you may think.) Besides to enjoy the material rewards of our ingenuity, we also see it as a moral imperative because it is a higher reflection of the same underlying value of building things to last that colours so much of what we have spoken of in computer science.

To achieve this goal we have to scope it and curate it somewhat for consumption by existing systems in our society. In particular, this means focussing on a particular chip design that comprises part of this system, the so-called Manitowoc microarchitecture, and transforming it modestly to be sold to the United States government. The design of these chips is highly unusual, but they are thousands of times more powerful (i.e. energy-efficient) at general-purpose computing than anything else on the market today. They approach the same order of magnitude of efficiency that tensor cores do, but with general-purpose computing instead of merely massive matrix multiplication.

Grovercomp will be built to provide a basis on top of which a Manitowoc chip can be realistically simulated. It’s just not possible to vindicate the designs without taking larger intellectual liberties on normal computers or FPGAs, but with Grovercomp in hand, we can approximate it much more accurately, just at a fraction of the speed. Upon running these simulations, we will take the results and apply for a grant programme with the Defense Advanced Research Projects Agency, otherwise known as DARPA. With a successful application in hand as a corporate resume, and the grant money applied towards a fabrication run at TSMC, we will attempt to sell these uniquely powerful chips to government organisations, in the spirit of DARPA itself.

The path forward from there is as clear as water, and as clear as mud. We will attempt to build the Anodyne computer spoken of in the original project pitch, and sell it to the general public with the productivity software in hand.

Thanks for reading! If you enjoy my work, please support me with a subscription to Nich Fury, my main newsletter, or via GitHub Sponsors. One-time donations can be made via PayPal. Until next time!

Subscribe now

Initially I tried to swab it off, and soon found this futile because there was simply no way I was going to get a thorough clean inside those heatsink fins by hand, even with a complete disassembly.

After a minute of outside-the-box thinking, I quickly realised the easiest thing to do is what was done to them with oil, but with alcohol – specifically, isopropyl 70%.

After all of the alcohol evaporated it left some considerable water volume behind. That’s okay, none of these things were powered and I was careful not to short them! But it will mean I need to sundry them for a day or two depending on how sunny and warm it gets here in the Piedmont autumn. We’ll take a look at that next time. 😉

Thanks for reading! If you enjoy my work, please support me with a subscription to Nich Fury, my main newsletter, or via GitHub Sponsors. One-time donations can be made via PayPal. Until next time!

Subscribe now

These are the first four (4) of what will ultimately be eighteen (18) cards deployed into the Grovercomp compute cluster! As before, the first two dented-up cards will be straightened out, repainted, and put into our personal rigs for more direct use.

Thanks for reading! If you enjoy my work, please support me with a subscription to Nich Fury, my main newsletter, or via GitHub Sponsors. One-time donations can be made via PayPal. Until next time!

Subscribe now

These two were had for $35 US apiece. One of them will be installed into my personal Linux rig, Henen-nesw, and the other will be kept in reserve for personal use by Charles Rosenbauer.

Here they are frontal:

Let’s take a closer look at these absolutely gorgeous backplates. This is machined aluminium!

I plan on hammering out a lot of the more grievous dents in these cards’ cover plates, and repainting them matte black with a screen-printed vintage Intel logo. Here is my mockup of what they will look like:

Thanks for reading! If you enjoy my work, please support me with a subscription to Nich Fury, my main newsletter, or via GitHub Sponsors. One-time donations can be made via PayPal. Until next time!

Subscribe now

Background

I am a computer programmer by trade, and a computer scientist by profession. So is my close colleague and partner-in-crime Charles Rosenbauer, and together we have been collaborating on various projects we share a sense of value in almost since we’ve known each other.

An abandoned server

For the longest time I used to work as an unarmed security guard, stationed at all sorts of industrial outposts around the Research Triangle as a literal watchman. (For the record, no, nobody watches the watchmen. I checked.) One of my last contracts was supervising an old decommissioned factory, and some of the junk they were having contractors throw away was a lot of dated server equipment, namely 10Gb and 40Gb managed Ethernet switches and a 24 TB storage server, which I came into possession of for a gracious $0. It’s amazing what people will throw away.

Gathering the Infinity Stones

Anyway, I saw a lot of utility in having these things, and in counting my blessings to find a relative lack of pressure to get a time-sink type of job, a gigabit fibre home internet connection, and a drive to work with Charles on things we really believe could change the face of computing forever, I decided to start looking into ways to get a hold of the only digital resource I lacked in spades for cheap: compute power.

Enter Xeon Phi

Very quickly I gravitated towards the now-defunct Xeon Phi computing platform, as I was just floored at how cheap some of these cards had become on sites like schmeBay. I bought two of them, and then four more, and then the other night bought 14 more in one go, at an average price of $40.50/card. All of them are the same model: the Haswell contemporary Knight’s Corner architecture, featuring 60 cores and 8 GiB of on-board RAM each; this is the SKU called the 5110P. Once again, it’s amazing what people will (nearly) throw away. Charles and I know that it’s an artefact of poor software that these things are so worthless to everyone else, and that’s exactly why they’re perfect for us.

The build plan

If you kept count, you know that I bought twenty cards in total. Two of them will be reserved each for Charles and I in personal rigs, while the other 18 will be deployed in triads among six compute nodes, which collectively make up the Grovercomp compute cluster. These nodes will be named after the first six letters of the Greek alphabet: Alpha, Beta, Gamma, Delta, Epsilon, and Zeta.

Specs & deets

While I have not nailed down specific model choices yet, as prices and stock fluctuate wildly, the general plan is to get something with these minimum specs for each of the six nodes:

• A dual-socket LGA 2011-3 motherboard with at least three double-wide PCIe slots for the coprocessors

• High core count Xeon E5 CPUs in both sockets, either v3 (22nm Haswell) or better yet v4 (14nm Broadwell)

• 8 GiB DDR4 ECC DIMMs ×8, simply to get quad channel memory with both sockets

• At least two 1kW+ Supermicro redundant PSUs

No over-engineering

That’s kind of it for the meat of things, really! There will be a fair amount of sysadmin work to be done once they are built, and I have no intention of over-engineering any of it. Will probably go with CentOS and harness resources together myself as needed. It’s actually a saner path when you write key code yourself already.

Utility

First of all, my napkin math shows me that at full load, this cluster can draw over 4 kilowatts of mains power. It goes without saying that I’m going to have to run lines to a few more additional breakers in my main service panel, and while I’m at it I’ll go ahead and add another one for my window air conditioner that’ll sit next to it, and maybe another for the rest of it all. Over-provisioning is a good thing, right?

It’s up to us

Okay, but more concretely, there are several things we can put this compute cluster to good use for. Odds are good that most, if not all, of these things are tasks only Charles and I have the cause to program in the way necessary to fully leverage the highly parallel and vectorised nature of the Xeon Phis. But that’s okay.

Einstein-level research

One of the research subjects Charles has been putting a lot of focus into lately is the SAT solver. Here is a screenshot he took of some preliminary results of a SAT instance, showing a 1024-bit lattice containing every possible arrangement of 10 bits:

Data science

Parallelisation is the pinnacle of scalable high-performance computing, and having this cluster enables us to experiment with algorithms at a scale normally unavailable to un-monied people like us. This system is just powerful enough and just complex enough (with 10Gb backplanes) that we can faithfully extrapolate performance factors for much, much larger compute clusters, up to even the world’s most powerful supercomputers, without taking gross liberties of interpretation of our data.

Software debugging and tooling

Even as just a single card in a personal rig, these Xeon Phi coprocessors also provide a fertile breeding ground for exploring new frontiers of debugging and software development tooling that, in Charles’ view, is woefully neglected. This is also a considerable portion of my interest in the Grovercomp: to create a workable software development toolset, principally anchored by a kernel (Hinterlib), an editor (Quindle), and an assembler-compiler (Oración/Gordian/FCC/Sirius) as the pillars of the toolset. Testing these on old dogs like my Compaq i486 is as crucial to their quality as testing them on machines that are as close as we may ever get to the state of the art.

This weblog

This place will serve as a kind-of-live feed of every major physical development activity that goes into the building of the Grovercomp, in the same fashion of the classic Groverhaus to which it owes its name and logo.

When it’s done, it’s done (mostly)

This publication will be done actively updating once the server is completely finished and operational, save for some potential occasional updates about outages, upgrades and anything else that may happen to it over its lifetime.

No paywall here

While I will not be paywalling any content here (this is Grovercomp, after all, so posterity is king), I do highly recommend giving my main newsletter a look-see if my content consistently interests you. The bulk of my writings there also revolve around computer science.

This is why we can’t have nice things

One of the biggest reasons I’m building such a strangely-architected computer is because, in a financial sense, I am quite poor, and cannot afford nice things like most people in tech. I’m actually due to ship back the M1 Max MBP I’m typing this up on back to my former employer, and it cost twice as much as this whole project will. I’m very gratefully taking advantage of a lot of non-material and non-monetary wealth I have earned in adulthood: my household, my family of roommates, and the collaborative effort we all take to make this house work so well in a way we could never do alone. Grovercomp is therefore my crazy pet project, and I will end up paying for it by selling kittens and donating plasma, if my strategy is right. Supporting me on Substack really helps.

Hold up, you said petascale!

I did, and I should be fair and say that our definition of “petascale” for this computer is in the context of 1-bit vector ops, not floating-point as is typical for measurements of compute power. Xeon Phi coprocessors have four-way hyperthreading which gives them four “hardware threads” per physical core, of which each card has 60. That totals out to 4,320 hardware threads from 1,080 cores, not counting all of the regular old Xeon cores and threads, of which we’re not sure how many cores there will be (we’re sure there will be 12 sockets filled though).

Thanks for reading and welcome to the Grovercomp weblog!

Subscribe now