Business Catalyst Blog

Incident report - downtime on AU data center

Cristinel Anastasoaie Tuesday, October 21, 2014


We’d like to give you an update on our recent stability issues.

As you know, as BC usage is growing and we continue to serve billions of page requests per month, we are in the process of continuously upgrading the underlying infrastructure for BC – from upgrading various operating systems, database versions, moving to SSD for faster performance and updating various other subsystems. As we’re also moving to deploy BC next, we are accelerating some of these upgrades to make sure next year catches us in a ready shape for more growth with no impact.

One of the areas where we have upgraded a subsystem was our load balancers – a pair of gateways to the whole BC datacenter designed to distribute load evenly on all web servers. Unfortunately, we are facing a load balancer bugs causing the underlying machine(s) to restart randomly, and consequently to produce downtime to the affected sites.

We have installed a premium load balancer with premium support, and been working closely both on our side and with the vendor to get this fixed. The vendor has identified the cause of instability in their code and promised to issue a patch for it.

However, given the critical situation we are facing, and given that we don’t have an ETA for the load balancer fix, we have decided double down our investment in this are that caused most of the recent incidents.

  • First, we have already added additional load balancer machines, so that if anything happens with one of them, we have a failover instance up and running, ready to take over.
  • Second, we are working on a fast failover mechanism which will minimize impact when a load balancer crashes down to 1-2 minutes – we plan to deploy this by Wednesday.
  • Third, we are also investigating the possibility of returning the currently upgraded load balancer software to the vendor and switching to a different provider, after rigorous testing.

And above all, we continue to stay alert to minimize any impact caused in the meantime, and we continue to keep you updated with our progress.

Another area where we’re making significant upgrades are the BC databases, where we’re moving to SSD and upgrading to the latest SQL server version.

We did finalize the upgrades in Sidney so far, and after one month of active load testing we can now continue deploying the upgrades in the EU and US datacenters.

For the EU datacenter, we are performing a database maintenance this Sunday, and this will be scheduled at an hour that will ensure minimal customer impact. The US datacenter will be upgraded late November.

Finally, we are working to make the BC Status Page monitors more accurate.

Today, these monitors do indeed show up as “green” if the majority of customers in a data center are up, even when some of the customers might have issues accessing their sites. This is caused by our “cell architecture”, where we split a datacenter in relatively independent cells in order to insure uptime in case one cell have problems for the rest of the sites.

We are changing the monitors to be more aggressive and show up as “yellow” or “red”, even when just a subset of the sites are down.

We really thank you for the patience you have shown with these incidents and we assure you we are doing our best to keep them under control. They are transient in nature as we’re finalizing this massive wave of upgrades needed for, and we are changing our processes to make sure we do test things more rigorously before deploying to minimize impact.

Thank You,
The Business Catalyst Team.

Thank you for joining us for Adobe MAX 2014!

Cristinel Anastasoaie Tuesday, October 14, 2014

We’d like to thank everyone who was able to join us in Los Angeles for Adobe MAX 2014! We had a great time seeing Partners at MAX as we could really feel the passion in the air at our Unconference and official sessions.

This time around we wanted to have a more informal setup that gathered the BC team and Partners together to shape the future of Business Catalyst. Therefore, we want to give a big thank you for taking the time to talk to us and share your thoughts on our latest releases and future plans. It has been great listening to you and we’re looking forward to bring your ideas to life.

While we have hundreds of comments and ideas, there are a few areas that will capture our attention next:

  • Sorting out the data centers stability issues – while most of the instability root problems (as described in our latest incident report) have been fixed, we’re still encountering some issues with our load balancers which we are working to fix as soon as possible;
  • Locking down the syntax so that everyone can take advantage of these updates; this includes items like unifying naming conventions between APIs and liquid variables and liquid syntax for module’s name parameters (see Lucian’s post on the developer forum)
  • Launch the final versions of and Open Platform, incorporating your top concerns/requests and focusing on training materials; the list includes items like: finishing the json output for modules, removing BC scripts and CSS from HTML output, updating the redactor editor to include support for image and link settings, finalise conversion to liquid for important modules (ex: module_subscriptions)
  • Start improving core modules once the priorities are validated with the larger Partner community and specific details are locked down; we’ll be having dedicated BC Townhall sessions for this subject in the upcoming weeks

Thanks again to everyone who was able to make it and share their ideas! If you weren't able to make it this year, we've got you covered as well. Below you'll find video recordings of all our sessions, we hope you enjoy them:

Looking forward for our Next event,

The Adobe Business Catalyst Team

App of the month: BC Client Training

Diana Herac Tuesday, October 14, 2014

Are you getting customer service calls that take up your valuable time? Are you interested in adding value to your hosting plans so you can charge more and make a residual income in the process?

Training & Automation are the answer. The BC Client Training App allows you to train your clients 24/7, reduces support calls, and gives you the opportunity to charge more for every client you put on the BC system.

Getting new clients and growing your BC business is challenging enough. One way to make sure that your hard work pays off is to train your clients on how to use the BC system and hooked on taking control of their online businesses as quickly as possible!

The BC Client Training App offers an always growing, up to date, library of video tutorials (64 videos at time of posting) that teach your clients the ins-and-outs of using the BC platform. Train your clients properly, make these videos available to them for reference (24/7) and watch the number of service calls drop, almost overnight!

All tutorials are recorded using professional voice talent, in both U.S. and Australian accents, and displayed using a sleek, modern, white-labeled video player that lets you call the training your own.

Have clients that don’t need training on E-commerce, or any of the other modules? Maybe you have videos of your own you want to add? No problem. You can customize which playlists appear for each client or add client-specific training of your own. Take your BC business to the next level with the BC Client Training App.

BC Client Training: Keep getting new clients. We'll help your train them.

The BC Client Training App is a partnership between BC Partners Scott Reynolds (Thrise/BC Sandpile) developer of the app and Greg Marcel (2140 Interactive/BC Client Training), content provider.

There are 3 videos in the playlist below. The first offers an overview of the App. The 2nd and 3rd are actual tutorials from the library, one in a US accent, the other in an Australian.

Incident report - recent downtime for AU data center

Cristinel Anastasoaie Wednesday, October 01, 2014

As you are probably aware, sites on all data centers have experienced some downtime these two weeks. First, we apologize for any inconvenience this might have caused you, and offer you a detailed explanation on what happened and what measures we're taking to prevent this in the future.

Starting on September 26th, sites on all data centers have begun experiencing intermittent downtimes. Sites on our Asia Pacific data center have experienced longer and more frequent downtime sessions than what we have announced in the AWS maintenance blog post.

The downtime has been caused by three distinct events and was amplified by timing:

  • Amazon AWS infrastructure upgrade - this operation implied many server restarts and failing over from one Amazon availability zone onto another and then back (basically, we had to execute a scheduled disaster recovery procedure). Our team has worked 24/7 to make this major AWS-wide infrastructure upgrade as smooth as possible to all our customers. During these procedures, the sites on the data center under maintenance became totally unavailable while sites on the other two data centers kept their front-ends running but had most of the back-end services disabled because we needed to stop the data replication between data centers. While Amazon has performed the restarts outside business hours for each region, the restarts of NA and Europe data centers fell during AU business hours and thus had some impact on all sites by preventing customers to access some of the back-end services. We are looking into implementing some architectural changes that will limit the impact of such operations from one data center to the other.
  • Load balancer crash - this week we have encountered a load balancer crash. We have worked with the vendor to identify the root cause and we decided to upgrade the system’s firmware; this procedure is almost completed now and we are closely monitoring the load balancer for any unforeseen issues that might arise.
  • A network connectivity issue between Amazon datacenters triggered an automatic fail over of the database servers to the backup servers. This type of operation usually generates a downtime of up to several minutes. We are currently trying to identify a potential network architecture change that could help mitigate this type of occurrence.

Once again, our apologies for any inconvenience this incident might have caused. Both our team and Amazon are fully committed to provide the upmost level of security and reliability to all our customers and we continuously dedicate efforts to improve on these fronts.


The Adobe Business Catalyst Team

Episode 40 of the BC Podcast - Interview with Alexandru Costin

Brent Weaver Thursday, September 25, 2014


With the upcoming BC.Next release, many questions are arising. What exactly does this update entail? What's the release schedule? What's next? Listen in as these questions and more are answered straight from Adobe's Senior Director of Business Catalyst, Alexandru Costin.

On this month's podcast Brent and I interview Adobe's Senior Director of Business Catalyst, Alexandru Costin on the upcoming BC.Next release, as well as what to expect at Adobe MAX 2014.

Listen in for your chance to win a BC Gurus template credit, valued at $149!

Listen to this month’s podcast: Simply click the play button below to stream the podcast via your browser.


Alternatively, you can download the podcast in .mp3 format for offline listening.

In this episode:

  • Brent and Adam interview Adobe's Senior Director of Business Catalyst, Alexandru Costin on upcoming BC.Next release and Adobe MAX 2014.

Join the Podcast!

Do you have something you want to announce to the BC community? We'd love to hear about it (

See you next month!
Adam Ladrach

Join Business Catalyst @ MAX 2014

Cristinel Anastasoaie Thursday, September 25, 2014

[Update: We've just released the full BC Unconference schedule and event details for Partner drinks on Sunday 5th October. Read the agenda below for more information]

We're happy to announce the Business Catalyst sessions and events at this year's Adobe MAX conference. This year is all about our latest releases and how partners can use these new features to deliver powerful more functionality for their customers. As always, attending MAX is a great opportunity to network with your fellow designers, developers and BC Partners.

View the agenda below, which includes a range of dedicated BC sessions and a BC Unconference catering to both new and experienced Partners. For session descriptions and times, check out the official Adobe MAX scheduler here.

Top Reasons to Attend MAX 2014

  1. Learn how you can get the most out of Business Catalyst Open Admin and BC.Next initiatives
  2. Get some hands-on training on liquid, new module syntax and REST APIs
  3. Expand your horizons and open up new business opportunities by networking with like-minded designers, developers and Partners
  4. Meet the team and have a drink on us - we'll be running a series of events where you can sit down with the people behind BC to discuss your thoughts and concerns.

To get your hands on a conference pass, make sure you visit the MAX website and register today!

Also, don't forget to attend our social event on Sunday 5 October. The event kicks off at 5:30pm and runs until late. Drinks and appetizers on us!

BC Sessions @ MAX 2014 Full Agenda

Sunday, 5 October

Monday, 6 October - The BC Unconference (fully booked)

  • 1:00pm - 1:15pm – Welcome & Introductions
  • 1:15pm - 1:55pm – Official Adobe opening presentation
  • 2:00pm - 2:55pm – BC.Next deep dive with Lucian Cozma
  • 3:00pm - 3:55pm – Open Platform deep dive with Radu Cosnita
  • 4:00pm - 4:55pm – BC Roadmap open table with Alexandru Costin
  • 5:00pm - 5:55pm – BC Sandpile with Scott Reynolds from Thrise
  • 6:00pm - 8:30pm – Meet the Teams, Adobe Booth in Community Pavilion

If you're unable to attend, you can also watch it live over Adobe Connect.

Tuesday, 7 October

  • 8:30am - Delivering Custom Web Applications Using Business Catalyst with Radu Cosnita

Wednesday, 8 October

  • 3:30pm - Creating Personalized Web Experiences with Business Catalyst with Lucian Cozma & Liam Dilley from Pretty

We look forward to seeing you at MAX 2014!

Scheduled system maintenance on all data centers - September 26th to October 1st 2014

Cristinel Anastasoaie Thursday, September 25, 2014

To ensure the highest levels of security and reliability, Amazon will be performing an infrastructure update starting on Friday, September 26th and lasting until October 1st. The update requires restarting all AWS hardware and will also impact the Business Catalyst sites. While we're taking pro-active actions to reduce the number of downtime sessions, we cannot completely avoid having some downtime for sites on every data center. Therefore, during the maintenance window, customers should expect a maximum of 4 downtime sessions of up to 3 minutes for every site.

Maintenance schedule:

  • Start date: Friday, September 26th
  • End date: Wednesday, October 1st

Customer impact:

  • Websites on all data centers will experience maximum 4 downtime sessions of up to 3 minutes

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

BC webinar: Pitching your competitive advantage with BC WebCommerce

Diana Herac Wednesday, September 17, 2014

This month we'll meet Wednesday, September 24, for a webinar that will teach you a thing or two on:

Pitching your competitive advantage with BC WebCommerce
- A love story about Clients and Elephants and offering the best value in town

Andy Mundell is owner of A NZ based web agency and BC partner who for the past six years has successfully sold all new clients on BC's full WebCommerce plan (including price driven entry-level customers) without meeting any pricing resistance. In fact, he argues that with the right presentation, WebCommerce becomes your compelling, competitive advantage. He shares his pitch including some valuable inclusions from OnCompany's proposal template, to show you how and why this is a winning formula.

Andy asks…

  • Is the cost of BC hosting the elephant in the room when pitching or pricing to prospective clients?
  • Do clients tell you they can get hosting for a fraction of the price elsewhere?
  • Do you bury or absorb hosting costs to compete with WordPress and other open-source based proposals?
  • Do you go for the lowest BC plan that will accommodate your clients requirements?

If you answer yes to any of these questions, then he says "you're doing it wrong and should attend this webinar".

You can register for this session here:

If you have any questions or suggestions please do not hesitate to send them over to

See you there!
The Adobe Business Catalyst Team

Scheduled system maintenance on AU and NA data centers - September 14th 2014

Cristinel Anastasoaie Friday, September 12, 2014

To ensure the highest levels of performance and reliability, we've scheduled a database server upgrade on our Australia and North America data centers. To minimize the customer impact, the upgrade is scheduled at the most convenient hours for the regions and will take up to 6 hours to complete. During the maintenance procedure, creating and updating content, Partner registration, trial site creation, publish from Muse, sFTP, APIs and some site admin sections will not be available for 4 hours on ALL data centers. Additionally, all sites on the AU and NA data center will experience up to 9 minutes downtime sometimes during the maintenance window. Except for the scheduled 9 minutes downtime, the website front-ends will not be impacted by the maintenance.

Maintenance schedule:

  • Start date and time: Sunday, September 14th, 12:00 PM UTC (check data center times)
  • Duration: We are targeting a 6 hours maintenance window

Customer impact:

  • Partner registration, Trial site creation Muse Publish, APIs, FTP and some admin section will not be available for 4 hours on all data centers
  • All websites and services on AU and NA data centers will experience up to 9 minutes downtime sometimes within the maintenance window
  • Creating or updating content on sites located on AU and NA data center will be unavailable during the maintenance procedure

For up to date information about system status, check the Business Catalyst System Status page. We apologize for any inconvenience caused by these service interruptions. Please make sure that your customers and team members are made aware of these important updates.

Thank you for your understanding and support,

The Adobe Business Catalyst Team

BC Sandpile September 2014 Schedule

Diana Herac Tuesday, September 02, 2014

Weekly Day/Time:

  • San Francisco: Tuesdays, 5pm (all times adjusted in relation to this time)
  • New York City: Tuesdays, 8pm
  • London: Wednesdays, 1am
  • Sydney: Wednesdays, 10am

BC Development: The Lazy Way
2nd September (3rd in AUS) 2014
Adam Ladrach from BC Gurus will show that even though developing sites on BC can be a complex process, given a little work, it can be much simpler. He will use his experience in managing development projects to show you some best practices in your development workflow, as well as managing the productivity of your team.

Open Meeting - September 2014
9th September (10th in AUS) 2014
If you were around when BC Sandpile first started you will remember that we were all about open and honest conversations focused on Business Catalyst. It was simple, valuable, and fun. We are going back to our roots with an open meeting. No presentations. No pitches. Just simple Partner-to-Partner discussions.

Solving Common Business Catalyst Problems
16th September (17th in AUS) 2014
Working with almost 200 BC partners and over 550 completed BC sites has given EZ-BC insight on what features and techniques are most important to BC Partners and their clients. Alex Yagolnik and Vlad Zaytsev of EZ-BC are going to share some of their best techniques, and answer as many Business Catalyst related questions as time allows.

Teaching Clients Marketing - the split training approach
23rd September (24th in AUS) 2014
Urszula Richards specializes in teaching business owners digital skills and marketing knowledge. If you have ever thought that despite your best efforts your clients are not quite getting it, then this presentation on the ‘split training’ approach is for you.

Full schedule:
Sandpile Recordings: