On Friday, July 7, an unauthorized connection to one of our technical partners resulted in the modification of the name servers [NS] of 751 domain names which then pointed traffic to the impacted domains to a malicious site.
Our technical team was notified shortly thereafter and proceeded immediately to reverse the updates on impacted domains and investigate the incident.
Pending the completion of our investigation, we published an initial report on the incident on our news site.
Now that we have completed these investigations, we can update you with further details about what happened and respond to several of the pressing concerns facing you, our customers.
Before diving into the details, though, we need to contextualize the incident:
At present, Gandi manages more than 2.1 million domain names across 730 TLDs, spanning some 200+ registries. Accreditation with each of these registries represents a significant contractual and complex technical component of managing these domains.
As such, we need to have the ability to connect to the technical back end of each of these registries, or otherwise integrate with a technical partner who connects in turn to each of these registries. Currently, we are integrated into more than 150 platforms, most run by individual registries themselves, others through technical partners.
The incident which occurred on Friday occurred only in regards to one such technical partner, through whom we manage domain names in 34 TLDs, all of which are country-code, and geographic TLDs.
Timeline of the events of July 7, 2017, in UTC (PDT is UTC-7)
- 11:57: A registry informs us of suspicious modifications
- 11:58: Gandi launches an internal investigation
- 11:59: Our team identifies a suspicious connection to our technical partner’s web portal
- 12:10: Our team changes the login credentials on our partner’s site, thereby blocking any further attack
- 12:30: Gandi begins working with our technical partner to identify all unauthorized updates made through their web portal
- 12:53: Gandi begins the process of undoing all unauthorized changes to name servers with each registry impacted
- 13:36: Our team completes the process of undoing all previously-identified unauthorized updates with each impacted registry
- 13:50: Our team begins working on reversing all changes identified in the meantime
- 14:00: Our team begins a parallel investigation in order to further ensure the security of our entire infrastructure
- 14:29: We publish our first public report of the incident to http://status.gandi.net/
- 15:00: Each of the individual registries begins to carry out the requested modifications on their servers; this update takes slightly longer for the TLDs .es and .se
- 15:32: We update our communication with the latest available information
- 16:00: Our team contacts the relevant French authorities
- 16:15: All registries have at this point completed all name server changes we requested
- 17:00: We are following several leads regarding the origin of the incident, but have not yet been able to draw conclusions with any certainty
- 17:50: At this point, we have confirmed with certainty that the unauthorized changes were made through the web interface of one of our technical partners
- 18:00: We launch a parallel investigation analyzing the malicious servers involved in order to better understand the impact of this incident
- 20:41: Our initial report on the incident is published in French.
- 20:42: Our team checks the Certificate Transparency logs for certificates issued on the impacted domains during the duration of the attack
- 21:23: Our initial report on the incident is published in English.
- 21:56: Our investigations confirm that the MX and SPF (TXT) records of each impacted domain were modified during the attack
- 23:02: The French authorities confirm that they have received our report
- 23:30: Our technical team launches a full security audit to ensure the integrity of our infrastructure
Some important points which to be aware of:
1. For how long were the affected domain name servers modified?
The first modification occurred at 8:04 UTC and the last was performed at 9:44 UTC. The last name server update was undone at 13:50 UTC.
Taking into account the delay in name server provisioning at the individual registries in question and the TTLs of the relevant DNS zones, the unauthorized changes were in place at the most for 8 to 11 hours.
By 16:15 UTC, all unauthorized updates we had reversed at each of the registries and we only needed to wait for propagation delay (up to three hours later) to be completely sure that the modifications had been successfully reversed.
2. What was the impact on the affected domain names?
The DNS servers (NS) were configured with A records for www.yourdomain.tld and yourdomain.tld. These records pointed toward malicious web servers.
Switch, the registry of .ch domains summarized the details of their analysis of the attacks and the exploits used.
Globally speaking, for the duration of the attack, visitors to impacted domains were redirected to an Exploit Kit (EK) type infrastructure, which rendered HTTPS traffic impossible. This type of infrastructure is capable of compromising the web browsers and operating systems of visitors in different ways according to:
– The intention of the attacker utilizing the EK
– The geo-localization of the visitor and the existing vulnerabilities in their browser or operating system.
The MX records were also configured with valid SPF entries, but according to the analysis done by scrt.ch, the mail servers to which they pointed were not functional.
3. What about SSL certificates on the impacted domains?
We also performed a verification of the Certificate Transparency ( https://en.wikipedia.org/wiki/Certificate_Transparency ) logs, cross-checking for any SSL certificates issued during the attack on any of the impacted domains.
We identified 18 certificates issued on domains during the incident. After a manual verification on each of these 18 certificates, we were able to conclude that all of these were legitimate since the owners of the domains in question all possessed the private keys to each of the 18 certificates issued.
Many organizations provide free services for requesting the Certificate Transparency logs, including:
- Facebook (which also lets you receive notifications whenever a certificate is signed for your domain)
- crt.sh (Note: using the syntax %.example.tld you can search all subdomains as well)
4. How was this attack even possible?
First of all, we should be clear that the attack did not involve any breach of our databases or back end nor did it involve a breach of our technical partner’s infrastructure. The attacker was able to make the changes by accessing the web portal of our technical partner using our login credentials, which they obtained surreptitiously.
These credentials were likewise not obtained by a breach of our systems and we strongly suspect they were obtained from an insecure connection to our technical partner’s web portal (the web platform in question allows access via HTTP).
As a rule, we have always systematically implemented all available security measures at all registries and technical partners (such as TOTP, IP restriction, etc.). Unfortunately, these security measures were only recently added, in 2016, by the technical partner in question and had not been identified at the date of our most recent security audit.
5. What additional measures has Gandi undertaken since this incident occurred?
All login credentials for all 150 technical platforms which we use to connect to registries and technical partners are currently being reset.
We have also launched a security audit on our entire internal and external infrastructure.
We also, as a rule, take advantage of the maximum security level offered by each registry or technical partner we connect to but we are now also in the process of double-checking that no new features have been added that we may use to further secure these connections.
6. Why did Gandi delay in communicating these details?
Our principal concern was putting into place all necessary and appropriate security measures (as noted in the timeline above) and immediately investigating the attack before the attacker was alerted to the fact that they had been detected. This unfortunately required a delay before we were able to publicly communicate regarding this incident, but this delay does not in any circumstance indicate that our team delayed in blocking the attacker or reversing the changes made.
We sincerely apologize that this incident occurred. Please be assured that our priority remains on the security of your data and that we will continue to protect your security and privacy in the face of ever-evolving threats.
