Earlier this year we faced a severe DDOS attack against our IP address that caused issues for some users accessing some of our domains. Mail service remained unaffected and webmail was also accessible through an alternate url, if the usual webmail site was unavailable. We apologize for the inconvenience and disruption following this unfortunate ‎incident. We have conducted a complete root cause analysis to find out what when wrong and how we can better deal with future attacks.

DNS Architecture

Our DNS servers are spread out over 16 IPs, in 4 data centers. Each server is isolated at both network level and physically, equipped with bandwidth capacity, network gear etc.

Client domains using our dns

Every domain name registered through Thexyz, a free managed dns hosting service is offered. Each domain is split into 4 name servers and the graphic below illustrates this.

‎Yourdomain.com is registered with us and gets 4 Name servers:

ns1.thexyz.com , ns2.thexyz.com, ns3.thexyz.com and ns4.thexyz.com.

Each name server has 4 dedicated IP addresses and in total, we serve our DNS traffic through 4 data centers, each with 2 physical servers which gives us a capacity of 16 GBps network throughput.

On each of these DNS servers we run a optimized version of PowerDNS with a capacity of 50000 qps. The total theoretical capacity of our DNS cluster is around 400,000 qps.

DDoS Mitigation Capacity

As mentioned before, our DNS servers are hosted at SteadFast and SteadFast’s network has been battle tested many times before during similar DDOS attacks. Each of the Data centers are equipped with multiple 10Gbps or 40 gbps transit links to the network. The data center also uses Arbor Peakflow for DDOS detection and Arbor TMS for DDOS mitigation. Each of the Arbor TMS systems are capable of mitigating 10+ gbps of attack traffic.
‎

‎What went wrong?

Usually when we see a DNS Server IP address getting attacked and they usually get null routed, it is often only attacking just a few of the 16 IP’s. This activity is pretty common and we see two or three such incidents every week. We have always maintained our service levels during all such incidents.

At its peak during the recent attack, we received 40+ gbps traffic spread out across all our 16 DNS server IP Addresses. The attack traffic was moving from one IP Address to the other at rapid succession. To prevent instability on the data center, they null routed our IP Addresses. The null route is a rule to drop all traffic destined to our IP address at the data center’s upstream internet service provider.

‎The problem with our setup

Issue 1: Relying solely on data center for DDOS mitigation capabilities.
Issue 2: We are bound to /32 static IP addresses. We are not utilizing our own /24 subnets to host the DNS servers. By using our own /24 subnets, we could have swung the traffic to our third party DDOS mitigation partner, Neustar.
Issue 3: All customer name servers point to the same IP addresses. So when attack happens and causes disruption for all customers using our DNS servers.

To solve these problems, we have planned a new DNS architecture and deployed this for use on our DNS services.

The new DNS architecture

Our new managed DNS infrastructure architecture is explained below:
We have moved the current DNS server IP Addresses to our own IP Subnets. This ensures we have the ability to use Neustar for DDOS mitigation when needed. All our data centers are already protected by Neustar. We will also start bucketing customers across different IP addresses so that an attack on a domain in one set, will not disrupt DNS service to customers in other sets.

We will start introducing local DNS services in other geographical regions where we have a data center presence and use Anycast. With Anycast, an attack originating from a particular region will only affect that region, while other regions remain unaffected. The affected region will use Neustar DDOS mitigation to mitigate the attack‎.

We sincerely regret and apologize for the inconvenience caused. We understand that you rely on us and to that effect, we’ll continue to render our services to the best of our ability to serve your business with utmost reliability.