Major issue on Gandi Mail

Mar 15, 2018  - written by  in Domain names

On Monday, March 12 a major network issue occurred on our Gandi Mail service which resulted in problems for our customers with sending and receiving email over the course of several hours. This incident has been resolved and Gandi Mail now functions normally. All emails sent during this incident have since been delivered, with the exception of those sent to Yahoo email addresses, where our servers have been temporarily blacklisted.

The underlying issue was related to a data center migration between our legacy data center, FR-SD2, and our new data centers in general. In particular, on Sunday we migrated the Gandi Mail platform off of FR-SD2.

Timeline:

Monday morning Paris time as a typically high volume of traffic arrived, we noticed a problem in the network architecture of the recently deployed servers: the network capacity was not sufficient to absorb the added Monday morning traffic.

As a result, this added traffic overwhelmed the entire network and rendered our other services unstable as well. Our teams first intervened to isolate this network traffic due to Gandi Mail usage so that we could keep it from taking down our other services.

Monday midday Paris time Gandi Mail was completely isolated but remained unstable.

At the same, our engineers reworked Gandi Mail’s network architecture in order to increase its capacity. At the beginning of the afternoon Paris time on Monday we were able to deploy this new architecture and the strain on the network began to decrease.

In order to help accelerate this process, we decided Monday afternoon Paris time to relax the rules on our email volume monitoring tool. This tool allows us to limit the impact of large spam campaigns regularly launched by botnets through our customers’ email accounts (without their knowledge).

This unfortunately resulted in inadvertently letting a spam campaign pass through and allowed a large volume of email to be sent by a botnet through our servers. We quickly realized what was going on, but not before several hundred thousand emails were sent, which resulted in several email service providers blocking email from our servers, specifically Yahoo and Microsoft (including Hotmail and Outlook).

Emails sent and received during this time were being added to a growing backlog in our email wait list.

Thanks to this wait list, emails that can’t be processed immediately can be held until such time as they can be. For example, if you send an email to somebody but the incoming email server for that person’s email service is not available, the email isn’t lost, but is put aside to be sent again later. Based on the number of times sending the email has failed, the email gets placed on lists with increasingly long wait times (that is, the frequency of each attempt to send the email goes down).

Due to this fact, emails sent during the incident were therefore taking longer to send even once service was restored than new emails sent after the incident was resolved.

This resulted in a situation midday Paris time Tuesday where some emails sent Monday morning Paris time during the incident were still not sent.

The current situation:

  • We have stabilized Gandi Mail’s servers; sending and receiving emails from either of our two hosted webmail services or from an email client, including emails containing attachments now works normally.
  • Emails sent to Yahoo email addresses, however, are still not delivered and our teams are actively working with Yahoo to get our servers unblocked.
  • All other email sent during the incident Monday have now been delivered.

If you are still running into problems with sending and receiving email, please use the form at help.gandi.net to report the problem, including whether the issue is with sending or receiving email, the software being used (an email client like Thunderbird or webmail), and the sender and receiver email addresses involved.

 

This incident occurred in the context of closing a legacy data center, FR-SD2, which we’ve communicated more in depth about previously. This was a major project for our technical teams that took months of preparation, including the migration of portions of all of our services, from Simple Hosting infrastructure, to Gandi Cloud, to Gandi Blog, and of course Gandi Mail.

The migration ends this week, with the final closure of FR-SD2. At that point, we’ll be able to take a step back and take what lessons we can from the difficulties encountered along the way and the impact those had on the quality of service provided and our own internal processes.

We would emphasize as a reminder that this was a necessary migration, required to get our services off of the now out-dated FR-SD2 data center infrastructure and onto our new infrastructure which will enable us to further evolve the services we offer. Our new data centers, FR-SD3, FR-SD4, and FR-SD5 will allow us, and by extension you, to take advantage of a superior network architecture, from new equipment, and allow us to improve overall performance on all of our services.

Of course, we are aware of the impact, in some cases severe, of these various issues on the quality of our services and we sincerely apologize for any inconvenience it may have caused you. Please rest assured that every team at Gandi is working hard to get us through the final steps of this transition and leave your services with us in the best shape possible.

Leave a Reply

    Our customers are not receiving our email, yahoo, aol, gmail we use outlook as our client, please help us

      Hi there, did you open a ticket as advised in our post? It is necessary for us to have a look on each customer account. Thanks.

    anonymous

    this whole migration thing from the old data center to the new data center, as well as from the old platform to new platform completely showed how gandi, a company that’s all about internet humanity lacks a strong technical back end to support their ambition and thus is very hard to cash their promise as “no bullshit” to it’s users (including me) again and again. And I just wanna say that I am not going to transfer my domain to another registrar any time soon, I am still satisfied about gandi being a domain name provider, but I am never going to use gandi’s email service again, I should have know free services just have their limits, lesson learned.

      Andrew

      Frankly, it’s not inaccurate to note that neither the data center migration nor the transition to the new v5 website have gone that smoothly. We know we could have done better. One of the main reasons that we’ve had issues is that both of these changes are upgrades that should have happened years ago.

      We’re making up for lost time and these two updates in particular are exactly how we’re working to get that strong technical back end that will help us better deliver on our ambitions, as you put it.

      Before you judge us too harshly (and I’m not saying you have), remember that even the best companies out there make mistakes and have massive outages. And a key way we want to be better than even them is how honest we are about our shortcomings and failures. That’s the essence of the “No bullshit,” motto; it’s not a promise that we or our services never fail, even spectacularly. Anyway, I’m glad you’ve decided to stick with us for your domain and if you’re committed to avoiding Gandi Mail in the near or long term, fair enough. We hope we can regain your trust as we continue to improve the mail service we offer.

      Just me

      I’ve reached the same conclusion actually. I’ve been a customer since 1999 or so, and seen Gandi’s ambition shape in recent years. And I fully support their ‘free Internet’ mission.

      However, looking at the size of Gandi’s team I really think their move to a, what looks like a very custom platform, based on non-standard technology that is developed in house, is maybe a bit too ambitious to develop and manage?

      Just me

      btw the mail service is paid, or has a paid option.

    William Gibson

    I am still experiancing issues with GandiMail, almost 1 week on. This is really annoying.

      Should be all ok now, did you open a ticket as suggested?

    Hi There,
    The contact form of my website (mails should arrive at my support@lipolytech.com mailbox) is not active and I still don’t receive any mail from the contact form.
    Help!

      Hi there, did you open a ticket about this issue already? Our customer care team needs more information to check what’s happening here.

    Matt

    As a long-time Gandi user, I applaud Gandi for taking the high road and publishing all their mistakes, outages, problems, etc. Several other hosted/cloud providers hide outage reports behind authenticated portals, don’t admit problems until forced, minimize known issues, etc.

    No one is perfect, every technical system will have outages and issues. That’s just a fact of life with modern complex computer systems. I appreciate how Gandi is forthcoming when problems happen, what’s being done about it and how to prevent in the future.

    You cannot scale and improve without sometimes accidentally breaking things. A dynamic hosted/cloud platform is always moving forward, up and onward. The key is learning from mistakes, preventing minor nits from becoming major outages, and continuous improvement/development.

    Gandi folks, please continue to keep your transparency and disclosure culture, despite any negative feedback.

    Sky

    Hang in there Gandi.
    There is no shame in reverting back to the legacy system. The old system seems to work pretty good. 🙂

    Nata

    Ticket # 9070244

    Please check it and advise

    Thanks

    Ted

    FYI: Outgoing mail server (SMTP) mail.gandi.net has issues this morning (22/03). Neither mail sent (attempted) from Thunderbird client, or webmail interface via SoGo, are successful.

    marco

    Well – what is the reason for today’s issues ? It seems that you are having a pretty poor March so far …

    Azat Fattakhov

    hi! is there anything similar happening right now?

    the emails that I send to my mailbox at Gandi’s are coming back with this message: “The email account that you tried to reach does not exist.” and I couldn’t find any reason for that in other posts or support page