We are continuing to experience intermittent issues on Adobe Business Catalyst's legacy Sydney datacenter. All dates/times in this post are in Australian Eastern Daylight Saving Time (+11GMT) We have systems engineers working on the issue. I know this is the 3rd business day in a row and it is really getting long in the tooth for Partners and site owners alike. Paul Gubbay, a VP of Engineering at Adobe will be posting on the blog shortly to share some of his thoughts on this very serious situation.
- Issue: Business Catalyst services hosted on the legacy Sydney Primus Datacenter are exhibiting slow response times. Webpages for these sites were being served slowly in addition to customers reporting problems accessing Admin UI or transferring data through FTP. This is the 3rd business day in a row this has occurred.
- Time of Incident Start: 2 Feb 2011 11:18AM Australian Eastern Daylight Time
- Time of Incident End: Ongoing - ETA is unknown
- Technical Action: Although we installed an extra switch and an extra firewall yesterday into the environment and moved OpenSRS migration to use the secondary switch/firewall, system engineers suspect there's still too much HTTP traffic coming through the primary firewall. I mentioned that we were going to put in a load balancer for the 2 firewalls as well but this was not required in the end because we put the second firewall on the second switch. Our plan now involves moving 2 web servers (out of 3) across to the network using the secondary firewall/switch combination to balance the load (resulting in 5-10 minutes downtime).
New Schedule For Migration
Onto the topic of datacenter migration; given all the feedback we've received in the comments below, we've now scheduled the migration to occur at 1:00AM Sunday 13 February (check local times here) to give the lowest customer impact possible for Australian businesses.
To give you some background, we originally chose 9am Saturday morning because we thought it would help those partners and site owners with externally hosted DNS make the switch in sync with the migration, however this isn't required anymore. Additionally you have all made it clear that the impact to your customers' businesses is unacceptable if we were to do the migration at the original time. With this in mind, the updated details are as follows:
- What's Happening?: We are migrating all sites and BC application infrastructure from Sydney Primus to Sydney Ultimo in one bulk-move
- Target Start Date/Time: 1:00AM Sunday 13 February 2011 (Australian EDT) | 6:00AM Saturday 12 February 2011 (US Pacific) | 2:00PM Saturday 12 February 2011 (London) | check local times here
- Target End Date/Time: 6:00AM Sunday 13 February 2011 (Australian EDT) | 11:00AM Saturday 12 February 2011 (US Pacific) | 7:00PM Saturday 12 February 2011 (London) | check local times here
- How Long Will It Take? We will have a scheduled maintenance window of 5 hours, during which all sites hosted on Sydney Primus will be unavailable. Partner Portal access and new site creation will be unavailable at this time as well.
- What are we doing? Simply put, we are going to replicate all databases between Sydney Primus and Sydney Ultimo. We will also setup a high-speed direct datalink between the 2 locations, to ensure databases are kept in sync prior to the migration. At the scheduled time of the migration we will reconfigure DNS settings and make other related BC architectural changes to point to the new Ultimo Datacenter. We will also need to restart all web servers.
- Customer Impact - Worldwide: During the migration you will not be able to create new BC sites on any datacenter. You will not be able to access the Partner Portal during the maintenance window. No action is required from you.
- Customer Impact - sites hosted on legacy Sydney DC with redelegated DNS: In addition to the above, all sites hosted on legacy Sydney DC will be offline for the maintenance window of 5 hours. There will be no front-end pages being served or Admin console access. No action is required from you.
- Customer Impact - sites hosted on legacy Sydney DC with externally hosted DNS: In addition to the 2 points above you will be required to change your DNS settings with your DNS host e.g MelbourneIT, GoDaddy etc, to point to the IP address of the new datacenter after the migration has started.
Sites with Externally Hosted DNS - Action Required
There's been some questions around what happens for sites with their DNS externally hosted. The Engineering team are looking into an improved solution right now which is to keep a proxy server in the legacy Sydney Datacenter so that all requests coming in to the old legacy IP addresses will get routed through to the new datacenter transparently. Likewise the pages being served will come from the new DC through the proxy server back out to customers. This is not a permanent solution but gives you a longer window in which to make your DNS changes and also lessens impact to your customers when you do make the change. We will likely keep the proxy server running for a minimum of at least 30 days after the migration before we fully decommission the legacy datacenter.
For partners or site owners with externally hosted DNS, we advise you to set the TTL for your records down to 1800 (30 minutes) during the next week in preparation for the migration so that when you do make an IP address change following the migration, the settings will take a shorter amount of time to propagateThanks for reading and check back in a bit for Paul's post.
Business Catalyst Product Manager