Network Time Foundation operates its own private cloud to support its computing and network infrastructure needs, both internal (development, testing, documentation, support ticketing) and external (email, website, code repositories, bug trackers, wikis).

Thanks to generous donations of equipment, space, bandwidth, power, and engineering support, the costs to maintain a private cloud are much less than equivalent public cloud hosting. I recently wrote a post about how running our own services is much more cost-effective.

Just like owning a car rather than continuously renting one, this does come with some downsides. One is that to keep costs low, in-house maintenance is a requirement. All equipment eventually ages and fails.

Additionally, NTF technical staff is comprised of software developers, not hardware folks, who are spread out across several continents, and I am the only person within driving distance of NTF’s Chicago data center. Therefore, it is important to be co-located at a facility with competent staff that can perform hands-on tasks. I am not always available, and it is imperative to have someone on-site if needed.

Deft, formerly ServerCentral, has been donating the space for NTF’s Chicago data center since 2016, and it is a great place to be. Nothing surprising happens. Power is reliable. Air temperature is stable and consistent. The network moves packets. The customer portal shows what is going on. Deft is great because everything just works, and has for years. It is hard to write anything particularly compelling when a provider’s baseline service is top-notch.

Then, one day something happened: one of NTF’s rack PDUs failed. It began as flapping of the PDU’s network port, which I had to shut down to quench the string of alerts. I had to visit the data center to install some new SSDs and replace a failed hard drive anyway, so I set out for Chicago. Upon arrival, I found that the PDU had dropped power to the gear. How fortunate that we have A/B power!

This was something I didn’t have an available spare for, and as I would be traveling the next few days, I wasn’t going to be able to remediate it later. Normally NTF might source used gear from data center recyclers, but that takes time. The other option was a new PDU, and I knew that $1000+ for a new PDU was not in the budget.

So, I went over to the Deft NOC and asked for help. A staffer immediately came to the rack, checked the breakers, and diagnosed the PDU as dead. I then asked if they had replacement PDUs, which they did, and asked how much. I mentioned that price was an issue, since NTF isn’t Deft’s normal commercial or enterprise customer.

The folks in the NOC took that and ran with it. They went and found a used PDU—the exact model that had failed—and got management to approve selling it for a fantastic (nearly eBay) price.

It took them 17 minutes to make the switch, including fetching the used PDU from a stock room, getting under the raised floor, disconnecting and pulling the failed PDU, installing the used PDU, and powering it up. The sheer competence, or should I say deftness, of it all was so refreshing.

Network Time Foundation and I are very thankful to the fine folks at Deft, not only for their handling of this issue, but also for their ongoing support of NTF’s mission which is:

“To cultivate and promote research, educational, funding, and development programs that keep the state of delivered accurate time in step with both the latest advances in technology and the needs of our consumers."