TLDR; Our domain faslet.me was migrated to Squarespace automatically, and DNS records broke, they haven’t been able to fix it and it’s been down for almost 2 weeks. Their customer service has not been helpful.
Introduction
It was the week that every Ops engineer has nightmares about. Our domain has been down for over a week and there’s nothing we can do about it.
What happened?
Squarespace bought Google Domains. Last week on Thursday, the transfer happened. At first I noticed that the SSL certificate for our domain was invalid. We have a relatively complex setup with some redirects, and no certificate was issued for our root domain, faslet.me, but instead, Squarespace had it fall back to their default certificate, which was causing certificate mismatch errors, and our page to show as “insecure”. That was the first problem. I reached out to their support via Twitter (chat support was and still is down), and was simply told the line “Squarespace does not issue SSL certificates for external sites”. Great, but you took this domain on, which had a certificate and attached your own? Again “Squarespace does not issue SSL certificates for external sites”. After a very frustrating conversation, eventually I asked to be escalated, they told me to contact support via the website, because they can’t help me.
More of the same
I sent in a usual support ticket, and got the reply after a few hours “Squarespace does not issue SSL certificates for external sites”. Things moved forward a little over a day or two, I managed to get access to the new DNS settings. Every time the support agent changed (several times a day), I got the same initial reply “Squarespace does not issue SSL certificates for external sites” and had to explain things again. At no point in the first 4 days did I feel like my problem was even acknowledged.
Problems Evolve
Eventually, out of frustration we decided to migrate our domain off of Squarespace. We run another domain (thankfully the one our API is on), on AWS without any issues. The initial plan was to switch the nameservers while the domain registrar transfer was in progress, to work around the issue. Initially all went well, email, SPF records, CNAMEs, etc. The final step was to edit/disable DNSSEC so that DNS servers that validate DNSSEC would propagate our domain. If you don’t do this, they won’t acknowledge your domain. Cloudflare, Google andOpenDNS all perform this lookup. This is where things went wrong.
Broken dashboard
We couldn’t modify the DNSSEC entry. Of course, being technical, we took a look to try to understand what was going wrong, but the server for their API calls was simply returning a 500 Internal Server error. Without modifying/deleting the DNSSEC entry, our domain wouldn’t be propagated on all the largest DNS servers. Those servers all return that our domain is non-existent. Ok, slight panic, but we can just revert the nameserver change right?
Wrong
That was also broken. Same 500 Internal Server errors when modifying those. In fact, the only thing we could modify in the dashboard was existing basic (TXT/CNAME) DNS records.
Passing the blame
We contacted support who told us they can’t do anything without Google verifying our ownership of this domain. I explained that they now owned the domain and Google could do nothing about this, and asked to be escalated to tier 2 support, as this had become more urgent. Again, Google needs to verify things, contact Google.
I contacted Google, who were incredibly helpful but basically said what I had already told Squarespace. They are no longer resellers of this domain or at all responsible for it, the transfer has happened and it is out of their hands. I went back to Squarespace with that, and was finally escalated to tier 2 support. To their credit they immediately saw that something was wrong, and offered to fix it, if I verified that I was an admin. I did this, and then another support agent change, again acknowledging the issue, and again escalating, this time thankfully to engineering. I was also told that there is no ETA and we just have to wait. This is where we are now. No resolution, the faslet.me is still down, emails are bouncing.
Damages
While we’re very lucky that this wasn’t the domain that our API is on (that is thankfully on AWS already), we are definitely losing business to this. We’ve received messages via LinkedIN asking if we’re still in business. We also presented at an event on Thursday, but couldn’t send people to our demo booking pages. This has almost certainly cost us tens of thousands of euros. And I know we’re not alone here (check r/squarespace, or the latest Trustpilot reviews). A few poor redditors had domains disappear on them! And the complaint is consistent, the customer service is awful.
Take-aways & Support training
Almost immediately, we discussed the support, and started using this as an example of what not to do.
Firstly, make your customer feel like they’re being listened to. Always acknowledge their problem. Even in the cases where they didn’t read the documentation or did something wrong, a major part of being a support engineer is to make the customer feel like you understand them and are on their side. I was made to feel like I was at fault for not reading documentation even though the documentation was not correct. Never do this.
Secondly, if a customer is technical, and your support is non-technical first line support, escalate them immediately if you can’t answer their query. My email thread with Squarespace is 42 (and counting) emails. At least 20 of them would be gone if I had been escalated on email number 3.
Now we’re in a waiting game. Thankfully on the weekend we were less likely to get buy-intended traffic (as a B2B offering), but if this isn’t resolved by Tuesday (tomorrow as of posting), we’re going to change our website domain and start over again. Thankfully we already own some alternatives, and just need to set up our website host properly. We’re also going to consult some legal experts to work out what compensation we can request from Squarespace for this. And hopefully gather people who have been similarly impacted by this so that they take this seriously.
