There’s been a fair amount of attention paid lately to the role of misconfigured NTP servers in the recent “network flooding attacks”. Much of the information in these reports is correct. Some parts are incomplete or inaccurate.
Some History
Over the centuries different groups of people have had the need for knowing the accurate time. Like anything else, people see what they can easily get. If they need better they put forth more-or-less reasonable effort to get it. One of the first places computers started to appear was in billing and accounting. If there was only one computer involved in this work, it was a bit less important if its time was right. It was a bit more important that the seconds ticked by at a proper rate. This is because in many cases the amount of time that passes is more important than the actual time. However, as computers were used in more and more places and more and more data was exchanged, the need to have the time synchronized on these computers became increasingly evident and important.
Network Time Protocol (NTP) is one of the earliest things to run over the Internet, and it is also one of the longest-running applications on the Internet. It’s 30 years' ongoing development and maintenance have been almost exclusively performed by volunteers. And not just “ordinary” programmers. We’re talking about two things here. One is folks that include kernel timekeeping- and network-layer software engineers, computer scientists, and physicists. The other is that these folks need to be, at least in my opinion, kind of eccentric or “different”. Slightly whacky, even. I mean, “normal” folks just don’t care that much about these things.
There’s another bunch of slightly strange folks out there, the systems and network engineers who “make things go”. They make sure the computers and networks stay up and running properly. One of the aspects of this work is keeping the time right, and NTP is the primary way that’s done. To do this properly requires that they be able to make sure the time servers they are using are in good shape. There are many real-world examples of other places this is done. We can see the health-department rating of the restaurants we go to. We can see the inspection certificate in the elevators we get in to. The planes we board are inspected by many folks at many levels, and the pilot or first-officer do a walk-around as part of their pre-flight checklist. Time is important - would you trust a clock if you had no way to know if it was running properly or not?
The Problem …
The opportunity for the problem comes from so many folks “playing nicely with each other” for the past 30 years' time, coupled with a general lack of “ingress” and “egress” filtering at the appropriate network borders, and/or NTP configurations that are “too open”. I just dove in to some technical stuff that some of you might not understand. There is a “border” in the network, between your (personal or corporate) computers and your internet provider, and similarly for every other internet-connected household and business. You have a range of “network addresses” you use - these addresses are used to send information to your computers. Pretty much just like the way you have and use postal addresses.
… and Some Solutions
The only information that should come in to your network from the outside should be from messages that never have your address as the “sender” - the only place such messages should come from are from within your network. An “ingress” filter at your border would block these forgeries.
Similarly, the only information that should leave your network should be from messages that only have your assigned (return) address as the “sender” - your network should never be sending messages that claim to be from somebody else’s network. An “egress” filter would block these. Better still, the information from an egress filter can alert you to problems like infected or cracked machines in your network.
If you are an ISP, you should be doing ingress filtering at your customer gateways. After all, their egress is your ingress.
While ingress and egress filters filters will stop outsiders from causing these problems in general, there’s also a quick and easy way to control what computers can ask NTP for information. That will also fix this particular problem.
The quick and easy solution for NTP is to add the noquery keyword to the restrict default lines in your NTP configuration file, in which case NTP will, by default, not answer query packets. Misused query packets are the “ammunition” in this attack.
That was difficult, huh?
If the fix is that easy, why isn’t it already in place?
Good question. There isn’t a really good answer, but here are some of the factors involved:
Ingress filtering helps “you”. Some folks will actually use this. But it won’t help much for the current abuse. Egress filtering will fix it, though.
Egress filtering helps “others”. Not many staffers are paid to make life better for other folks, and egress filtering is not compulsory.
Your ISP could do ingress filtering at their network border with you. After all, your egress is their ingress.
This is the first time this NTP facility has been abused in a major way in over 20 years' time. The potential for abusing this facility has been known for a very long time and there has always been a way to block it, but changing the default from “open” to “closed” has had tangible costs and has never had tangible benefits before.
Filtering, by design, “limits” the flow of information, and therefore if a situation ever arises where we need to “poke a hole” in these walls it is extra work. If there are no walls, there is no need to poke holes.
There’s a better fix for NTP..
… and it’s been sitting in the NTP codebase since 2010/04/24, and that fix will be in ntp-4.2.8, the next production release.
What’s the delay? A few remaining critical bugs. And remember, nobody is being paid to work on this. Yet. Also, there are huge numbers of computers out there that are still running versions of NTP that are 5 to 10 releases old, going back 20 years! Given the current state of things, filtering (egress for you, ingress for your ISP) is still our best answer for this as well as related attack prevention.
You can help make that fix, and others, happen.
NTP development and support has, to this date, been a volunteer effort. We could do a lot better if we could pay qualified people to work on it. Is having the correct time on your computer worth $1/year to you? More? Less? Join the NTP Consortium at Network Time Foundation, or support our efforts by making a donation!
Folks who want to make sure their NTP configurations are up-to-par can also visit https://support.ntp.org/Support/ConfiguringNTP for significantly more thorough coverage of this issue.
Until the Network Time Foundation came to be, there was no base from which to run efforts to spur manufacturers and consumers into keeping their NTP servers up-to-date and properly configured. Now, NTF is working on a Certification and Compliance program that will develop “Best Current Practice” documents (BCPs) for network timekeeping and establish base levels for acceptable use and behavior. Think of it as a way to keep your company off the NTP Vandalism pages. If this project is something your company is interested in, please contact us.