DDoS
Incident Report for Gandi.net
Postmortem

*Update about the January 23rd DDoS.
*

TL; DR :

It was the biggest DDoS attack  we have ever faced.

The attack lasted from 06:25 AM CET to 12:56 PM CET

Services were unavailable or degraded from 06:32 AM CET to ~ 11:00 AM CET

At 06:42 AM CET :

Our public services monitoring was triggered by external probes.

The whole infrastructure team was paged.

The monitoring was very noisy due to the DDoS. Metrics were also degraded, making it difficult to find the target.

After further digging, we saw that several public services were under attack.

We had to isolate and defend several service endpoints at the same time.

The fact that the attackers were targeting different services made the attack more complicated to mitigate.

Adding to the difficulties, our backbone and some core links were saturated, making troubleshooting and remediation slower and more complicated even with out of band connections.

We lost  a lot of time on that.

At 7:52 AM CET: We were joined by our anti DDoS provider team.

We were beginning to advertise some service prefixes via our anti DDoS provider and start mitigation while isolating those same prefixes from our standard transit and peering providers.

Our priority was to defend our DNS servers.

The DNS cluster was doing well. However, as network pipes were full, the service was degraded.

Other endpoints like admin.gandi.net and public APIs were still down.

We had some difficulties in deploying efficient mitigations due to technical problems on our side while trying to advertise prefixes only through our anti DDoS provider.

At 11:00 AM CET our services were mostly back online.

Our internal postmortem is ongoing to see what went well  and what could be improved, and there is room for improvement.

Posted Jan 27, 2021 - 09:42 UTC

Resolved
This incident has been resolved.
Posted Jan 23, 2021 - 18:15 UTC
Update
We are continuing to monitor for any further issues.
Posted Jan 23, 2021 - 12:02 UTC
Monitoring
Situation is stable for now. We keep it under monitoring.
Posted Jan 23, 2021 - 12:02 UTC
Update
DDoS is still on going. Services are back to normal for most part of it. Team is still working to adapt mitigations.
Posted Jan 23, 2021 - 11:24 UTC
Update
DDoS is still on going, we are working to improve mitigations to keep services stable. You may see some services disruption again.
Posted Jan 23, 2021 - 10:23 UTC
Update
Services are getting back to normal, mitigation is in place, we are still tracing some remaining saturations problems
Posted Jan 23, 2021 - 09:59 UTC
Update
We are isolating services to bring them back up one by one.
Posted Jan 23, 2021 - 09:28 UTC
Identified
We are still targeted by a DDoS attack.
Our teams are starting to push mitigations.
Posted Jan 23, 2021 - 09:24 UTC
Update
We are still working to mitigate the attack.
Posted Jan 23, 2021 - 07:50 UTC
Investigating
We are under DDoS, teams are working to mitigate the attack.
Posted Jan 23, 2021 - 06:14 UTC
This incident affected: Domain name registration, Network, SSL Certificates, Hosting (PAAS FR-SD3, PAAS FR-SD5, PAAS FR-SD6, IAAS FR-SD3, IAAS FR-SD5, IAAS FR-SD6, PAAS LU-BI1, IAAS LU-BI1), Portal (www.gandi.net, Billing / payments, admin.gandi.net), and Gandimail (SMTP in, SMTP out, Roundcube webmail, Sogo webmail).