• Team
  • Team

GitHub & the memcached DDoS amplification attack of 1.35 Tb/s

GitHub survived a DDoS attack of 1.35 TB/s. How?
By Yves Junqueira

March 23, 2018

GitHub survived a DDoS attack of 1.35 TB/s. How? They paid someone else to figure it out, as they should.

If you leave your memcached servers listening on UDP and open to the Internet, they will certainly be used to amplify DDoS attacks. During the event described in the article, which happened on February 28th, GitHub had to move all their incoming traffic to Akamai so they would handle the attack – 1.35 Tb/s.

Don’t be an amplifier

Remember: unless you know what you’re doing, do not leave UDP services open to the internet. Most UDP services out there (including NTP, DNS and even memcached) can be used in amplification attacks.

Amplification attacks are simple. The attacker forges a packet pretending that it’s coming from the victim, and sends it to a UDP server. When the server responds, it ends up sending a potentially large response to the victim. The “amplification” happens because the attacker sends X bytes to the server but the server responds with Y (where Y > X, usually multiple times larger), meaning the attacker can use 1 byte of traffic to generate X bytes of untraceable packets.

For example, imagine the attacker asks “hey memcached, give me the object with this hash key”, where this is a field with 8 to 200 bytes. If the attacker requests an object that exists, memcached will respond with a much larger object of, potentially, several MBs.

Since people that put their memcached on the internet usually don’t care about authentication, the attacker can cheat. They can set up the attack by inserting arbitrarily large keys into the cache, and then ask for the object again – except they would forge the packet to use the victim’s address.

By repeating that process many times per second, the attacker makes a random memcached server send hundreds of MB/s of traffic to the victim. And that’s from one node. Now assume the attacker has a list of thousands of open memcached UDP servers…

Very nasty.

CDNs are a must

CDNs with DDoS absorbing capabilities are a must for all businesses connected to the internet.

Sadly, we really can’t make your own CDN, not one that’s useful for absorbing these kinds of attacks.

Given the internet’s architecture, Do-It-Yourself DDoS blockers are just not effective. There’s no way to stop that traffic from flowing into your network unless you ask other peers in the network to absorb your traffic.

But maybe that’s one way? Wouldn’t it be nice if users and companies could help each other absorb DDoS attacks, in exchange for getting similar protection? That’s not easily possible for most users (e.g: residential) because they can’t announce BGP from their networks.

The next idea would be to organize BGP-empowered companies to help each other absorb DDoS attacks.

Sounds good on paper. In practice, it’s better for companies to focus on their business, and pay for Akamai, CloudFlare, etc to worry about DDoS for them.

Thank you capitalism?

By Yves Junqueira
Yves worked for a decade as a Site Reliability Engineer at Google, where he created software development tools and infrastructure. He left Google in 2017 to found YourBase, where he is using his passion and experience to help development teams move faster. Follow @cetico on Twitter.

Recent Posts

YourBase Early Access - software development without distraction

October 30, 2019

Four years ago, my family and I moved from Zurich to Seattle. I grew up in Brazil and had the…

Benefits and downsides of microservices

March 15, 2019

The main benefit of microservices is to allow a large team of engineers to development software more…

YourBase - let’s fundamentally simplify how we deploy software

January 23, 2019

YourBase is a community of SREs and developers from different companies that are starting a new…

GitHub & the memcached DDoS amplification attack of 1.35 Tb/s

March 23, 2018

If you leave your memcached servers listening on UDP and open to the Internet, they will certainly…

When should you create new microservices?

January 12, 2018

Microservices are somewhat like camping: the first time you do it, there is a lot of uncertainty and…

We need base services and not just microservices

January 02, 2018

I wrote recently about how the monolith slows down teams with more than 20 or so developers and why…

LoginDocumentationHow It Works