My cell phone cuts out for awhile during my drive to work, I lost
the
first draft of this blog posting when I accidentally pressed the wrong
button, and I could probably use up most of my free time if I fixed all
of my friends’ and relatives’ blue screens, spyware
infestations, and other computer woes. I can’t stand it! But it amazes
me that most people seem to accept these inconveniences as part of life
in the high-tech world.

I don’t think that’s right — we need to expect more from our
high-tech products.  A few companies do deliver great service
(what was the last time you saw Google down?) but most seem to have an
attitude that a little downtime is okay. For example, my main financial
web site takes its systems offline for hours of maintenance on Sunday
nights — just the time I finally get around to taking care of my
financial health.

I don’t accept this for QuickBase. QuickBase should be "always on",
available with your data when you need it. For example, if your team
meeting presentations are stored in QuickBase, you can’t afford to have
it go offline in the middle of your meeting.

We think a lot about QuickBase reliability and performance. I hope
you don’t. I hope you expect it to "just work"; even in the middle of
the night when you’re entering your latest inspiration into your
QuickBase To Do List (you do use QuickBase to manage your task list,
don’t you?)

Over the last decade, I’ve developed Internet services using
thousands of servers and supporting hundreds of major web sites. Here are my 5 keys to creating successful 7×24 (7 days/week,
24 hours/day) services. We apply all of them here, day in and day out.

1. No Tolerance for Downtime
   You can’t fix a problem
   until you accept that you have a problem. Your whole organization from
   senior management through product developers to data center operators must detest
   downtime. Must detest it more than their customers do. You need that kind
   of emotion to get you out of bed when the pager goes off at 3AM.
   A
   few years ago, when we needed to restart QuickBase for a software
   patch, it was offline for 10 minutes. Pretty good, but just not
   acceptable to us, so we reduced it to less than a minute. Still too
   long. As of our June release, most software upgrades can be done with
   no downtime at all.

2. Test, Test, Test (aka Don’t Trust the Vendors)
   Nothing works the first time. Okay, that’s not always true but it’s a
   good approximation to the truth. During my time with QuickBase, I think
   we’ve endured failures to every single one of our "highly available"
   and redundant pieces of equipment. One dirty secret of "high
   availability" systems, is that the additional redundancy often just
   adds complexity and more ways that things can break.
   The only way to prevent this is to test every system with realistic
   loads and (temporarily) break each component to see how the system reacts. We do this
   on everything from the power grid (it’s quite impressive to hear the
   semi-truck-sized diesel generators start up when the utility power is
   shut off!), to servers, to network, to each software release. Before you see a new software release from us, we’ve already subjected it to full production-level loads.   

    

3. Log Everything
   Stuff happens… However, you have to make sure that you know it happened and how it occurred. Good system status logging saves a lot of time and money: the faster you can identify a problem, the faster you can fix it.
   QuickBase has the best logging of any web service I’ve seen. We’ve made it easy for the developers to add diagnostic logging into QuickBase without slowing performance. Because of this, we usually know about problems and potential problems long before they affect customers.
    
4. Efficient Crash/Problem Recovery
   When the rare problem does occur, we’ve worked to insure QuickBase minimizes its impact, and records the information we need to quickly find and fix the problem (including the exact source code line number for software problems). Although we prefer to have our operators and developers focused on building out ever faster servers and adding more new features they are a great team in a crisis and focus on safely restoring service ASAP. They save the "could’ves" and "should’ves" for later, at the "Lessons Learned" sessions we hold to review each significant outage.
       
5. No Single Point of Failure
   While the above keys set the stage for a reliable service, a successful 7×24 architecture must avoid any single point of failure. One of my jobs is to regularly review our architecture to make sure that ANY component can fail without compromising QuickBase’s reliability.
   We don’t have one of anything in QuickBase and we generally have more than two. Everything is backed up from servers, to power cords, to network connections, to storage, to CPUs, to our diesel generators.

I hope you expect a lot from us! We expect a lot of ourselves.

– Jim

P.S., Our Operations manager will hate this post. Every time I brag about QuickBase’s reliability, disaster seems to befall us. So if QuickBase goes down in the next few days, it’s my
fault.

Sorry!

You’ve already been introduced to some of my colleagues. I’m Robert, the new kid on the block who was recently hired as the Product Manager for QuickBase. The first 30 days have flown by, and I am so excited to post my first Blog entry.

Last week the sun finally came out in Boston! To celebrate, my good friend hosted a BBQ party on Sunday. Lots of of people came, including some of his neighbors that I didn’t know. As these things go, it wasn’t long until one of them came up and asked the inevitable question …

"So Robert, what do you do for work?"

In my excitement, I began to tell him about QuickBase. Literally, 15 minutes later, my wife came up to me and signaled that we had to go, yet I was only 10% of the way through telling him about QuickBase and how many amazing solutions our customers have created. This got me thinking…How do you summarize a product as unique as QuickBase in a couple of sentences or less?

And so this Blog entry was born, because I would love to hear from you….How do you describe QuickBase to your colleagues, friends and family? Looking forward to your insights.

My current definition of Business Utopia: IT and business folks working together to build solutions to real problems with appropriate technology. I see so much good technology gone bad, because, IMHO, these two factions aren’t translating between their different languages. Our example… Many prospects and customers don’t want IT involved in their QuickBase account. Their top reasons:

1.    IT will take control and not let me do what I need.

2.    IT will take too long to build a solution that then won’t work.

The flip side, IT customers often don’t let their business friends use “their” QuickBase, because:

1.    They will not try to understand it; I’ll be stuck supporting their mess.

2.    They will use it for everything; I won’t be able to manage the potential growth.

Both of these parties are losing out on helpful, even critical, input and support with these positions.

They can get what they want though. Example… a top financial services company using QuickBase for event management, which is not core to their business offering, but it is necessary for their business operations. Together, IT and the business users had spent $300,000+ on an event management system that never worked. IT had to field requests to change reports “all the time”. The business user had to wait two weeks to for a new column on a report that they needed “yesterday”. Using QuickBase, IT has the standardization, centralization, and security oversight that they need for corporate applications, and the business user has the immediacy and “exact solution” they need. Both sides understand each others needs now.

Since we continue to find it a challenge to get the business folks and the IT folks to talk together about solutions, do I have my head in the clouds thinking QuickBase can help?

P.S. Yes, I’m back. Sorry for the delay. I’ll respond to the recent (great!) comments this week; I had to get these thoughts off my chest… even during budgeting we had the talk about "avoiding IT" as a benefit for QuickBase; I don’t like that! Oh, and… drum roll… FY06 Budgeting review went really well! Thanks for all your support. YAY!!!