To our Customers – Response from the CIO

We’ve received a lot of feedback from our customers about the power outage and the way we communicated with you. I appreciate the messages of support and I understand those expressing frustration. Though it doesn’t happen often, when there is a service disruption, we know it’s vitally important to keep the lines of communication with our customers open. As you may know, this outage affected many Intuit products and services across the entire enterprise, not just QuickBase, so finding out and sharing information took longer than you have come to expect from us.

To help address your concerns, Ginny Lee, the CIO of Intuit, has shared additional information below about what happened and what we’re doing about it.

I understand that you place a great deal of trust in QuickBase and it’s an important component to help you run your business.  I personally feel devastated by this event and my team and our company shares these feelings.  We take tremendous pride in our product and we don’t take unanticipated setbacks like this lightly. I want you to know that we aren’t going to waste this opportunity to learn and we are committed to moving forward and making our product and services better than ever and to regain the trust and confidence you have in QuickBase.

A first step in that process is to open the lines of communication even more. We are available to help you through our support center and we’re reaching out to those of you who have entered cases.

We’ve also added another option to reach us with questions, comments or concerns. Please feel free to contact us at 1-781-370-4414. This number will be available until July 1, 2010 and will be staffed from 9:00 a.m. – 5:00 p.m. Eastern time.

Thank you for your patience and understanding.

Rhonda

Note from Ginny Lee, Intuit CIO

To our customers:

We understand we didn’t communicate as frequently as you or we would have wanted. This issue was complicated and involved a number of Intuit applications.  We did not want to share any information until we had complete confidence in it.  

We realize now that in times of uncertainty it is better to share updates on our progress as they occur versus waiting for complete answers, not sharing enough and leaving you feeling in the dark.

We still have unanswered questions, but we do know more and I want to share it with you.

We had an accidental power failure that brought both our primary and secondary power units down. The process of restoring power caused a hardware system failure. We still don’t know the root cause of that. To ensure full integrity across all our impacted applications, we immediately initiated a restoration process using our back up data.   We also had to ensure proper sequencing in a way where both individual applications and our own enterprise applications that they rely on are restored in the most reliable and secure way  before bringing them back on line. This took more time than anyone would like, but it was the safest most reliable course of action.

We will continue to look into both the root cause of the failure as well as our response so that we learn from this and get better. We appreciate your patience in what has been, understandably, a very frustrating experience.

Ginny Lee

Chief Information Officer

Posted in People Management | Tagged
  • Francis

    Rhonda — I am a former customer of Quickbase (former only because of the loss of the contract that required your service, not because of the service).

    We had one 1-day outage in the two years we were Quickbase users and your company communicated extremely well during that problem time. Quickbase is a wonderful product. Please keep leading the way for cloud products and customer service. In my view, the profit from that leadership is realized in the future, when this work-model becomes commonplace. Machines break, they always have, they always will, you guys handle it better than anyone. Customer's frustration, even anger, is understandable, but don't let the crazies guide your vision.

  • Mike Grier

    I'm an IT provider, and as such, I can sympathize with what the Intuit team went through to get all systems back up and running. Although we use QuickBaseconstantly and were negatively affected, I feel certain that all that could be done was done. Systems can fail at any time, whether they be Intuit's, Google's, Zoho's or my own in-house systems. Those seeking Nirvana – let me know when you find it. :-)

  • Nranelli

    I'd like to understand you infrastructure, are you only in 1 data center…I'm curious on your CIO's note to us, if you lose power both primary and backup where is your Disaster Recovery…do you have a plan that move your customers data to another location with this type of event?

  • Chris

    Ms. Lee,
    I appreciate the Note, and what you must be experiencing right now. I also feel compelled to question how it is that the backup data center wasn't employed, thereby providing that level of stability and advertised service we, your consumers, were lead to expect in a Platform as a Service provider. I think the issue you're hearing from QuickBase consumers isn't that there was an outage, but WHY there was such a poor architecture in place to mitigate the impact to our operations and Mission. It is my hope that future Notes from your office address this in the form of a “Get Well Plan” that will honestly address what your customers perceive are the true issues. Outages happen, it's the response to and reason for the outage that is at the heart of the matter.

    Thanks

  • Tshackelford

    I had a significant impact to the productivity of my help desk. They received calls all day and couldn't enter any case information into Quickbase. We're still being impacted today as they are trying to catch up on the documenting cases. How are we going to be compensated?

  • PBaer

    Ginny, it would have helped us keep calm if we knew that the cause of the outage was a power failure rather than an application issue or some other disaster scenario.

  • Sjohnson

    Funny, I thought there was plenty of regular communication during this outage.

  • Joe

    I'd like to know how often you test your disaster recovery processes. Additionally, to echo a previous comment, how is it that you didn't have an alternate data center to switch to?

  • Jessica White

    This is moving in the right direction, but I still have questions, and serious concerns about the reliability of Quickbase.
    1. What do you mean by “accidental power failure”?
    2. Why do you have your critical infrastructure in one physical location? I heartily agree with many commenters who believe that this is not the level of fault tolerance one expects from a SaaS provider.

    Please respect our need for more information. I have to make decisions about where to store my company's data, and unless we learn more, I'm seriously considering bringing my database in-house, where I know it will be well taken care of.

  • Chris Garris

    Thank you for the additional information. I, too, was extremely frustrated by the lack of technical information during the outage. Sincere apologies are nice, but in this age of instant availability and access to news from across the globe, it was inconceivable that more data regarding such a significant service failure by a major provider was not to be found. I totally discounted the rumor I saw about a power outage being the cause – I had total faith that Intuit had so many redundancies that this could never happen. Certainly an outage due to a power issue would have warranted a simple statement of fact and not the “stealth mode” we encountered. Lesson learned…even Goliath was taken down by a pebble.

  • gb

    My understanding is that Intuit has multiple data centers cross the county. Why was it that you could not fail over to one of those DC? Why did you have primary and seconday systems located/powered in one DC? If this was caused by a power failure in on location, what would have happened if there was a major disaster (like an earthquake) that wipes out your DC? I would like to have a better understanding of your DR plan to ensure cloud uptime and assurances.

  • Chuankai

    The product is still not fully functional: I cannot promote a user as a manager even if the current manager is on deny list.

  • Rob

    It sounds like, moving forward, this should be an easily avoidable situation – if lessons have been learned and action taken and followed through with.

    I fully support the quickbase team and love the service, even in spite of this downtime. I'm glad none of my data was lost.

    None of us, however, can withstand many repeats of this situation, so I hope action is taken to reengineer so this would not happen again. Would be a great boost to confidence if a sanitized and secure version of the fix was published.

  • Doug

    Thanks for the feedback Ms Lee. It sounds like you did not have sufficient UPS and Power Surge Protection. True?

  • Private

    I am really not impressed. With lack of openness, lack of technical details, the appearance that one key component trigger the cascading failure and the fact it took 24 hours to resume service.

  • QB_Support_Phil

    Chuankai,

    My name is Phil Powis and I am part of the support team here at QuickBase. I attempted to reach you by phone with the number associated with your account but was unsuccessful. I have also opened up a support case in regards to your issue. You should have already received a notice of this. I would be glad to help you resolve this issue.

  • Patti

    Will this affect/postpone the scheduled Infrastructure changes that are set to take place June 26?

  • Tom

    To everyone at Quicbase
    I have been a customer of Quickbase since the inception of the program. My experience with Quickbase has been one of the most trusted with all of the “cloud” systems that our companies use. I have read everyone's frustrations and comments of what should have been done, which will be I am sure taken into consideration by the Quickbase team. Though the outage was a nuisance, we should all be responsible for our companies. I back up my data consistently and therefore even though all of our users were not able to access the “cloud”, our company moved forward. I commend Quickbase for their actions, their notices and would recommend to all businesses to always prepare for the unexpected.

  • L Gray

    QuickBase is indeed an excellent service. That's not at issue.

    The issues are:
    (a) Users were told only recently of the nature of the problem, i.e., an “accidental” power failure, and
    (b) Assuming that's true, how does one power failure take down both the primary and backup systems?

    To my knowledge, neither of those issues have been adequately addressed.

  • LGV

    Ms. Lee, Rhonda;

    OK, how many data centers are there *really*?
    Can you fail-over switch to them when one data center turns to toast?

    When I called years ago asking for official statements on security, backup, reliability, I was told you have multiple DCs with failover. How is that explained now?

    What is the structure of the DCs when you switchover June 26th?

    Thank you,
    LGV

  • Jreicher

    How does the primary power fail and the backup power fail at the same time. That's not much of a backup system to me?

  • Edgardo F Herrera

    Mike; The problem is not what was done to recover. The problem is was not done to prevent it. These types of online service systems must be secured with redundant power, servers, and even facilities. Their communiques indicate Intuit has all their product servers in a single facility, all fed from the same power source. A complete (main and backup) power failure just stopped ops for a full day. The power outage could have been just 30 munites. Restoring the systems is what took most of the time.

    I get questionaires from potential clients that demand assurances of security and safety of their data through our services. In this day, this is all minimum standard installations for this type of service. The damage in our case is intangible as erroded confidence can have effects that will be felt after long replanning sessions and service procurement process by our clients.

  • Edward Bianco

    As a CIO, I understand the issue but can’t sympathize at all. I would have been on the street the next day if I would have allowed my organization to fail in such a preventable manner. A company of your size and resources and marketing your product as outsource SAAS solution, this is totally unacceptable and someone in the organization must be held accountable and fired if they did not perform and test their DR plan. At every firm I managed IT, DR was woven into everything we do and we tested and tested and tested our plans including scenarios such as worst case such as double and triple failures in core services/infrastructure and ability to recovery all services at HOT recovery site. Today we can recover our core systems and applications within 5 minutes using Double Take real-time replication to our farms of Vmware clusters. As part of this we regularly test our Business Continuity and Disaster Communication plan to we can reactive to the disaster. It seems that you failed in all areas. to be prepared to act.

    As an Enterprise account with over 100 concurrent users using our custom built applications “Critical Systems” we are more than just disappointed we are angry. We lost over 800 hours or productivity for a service that was sold as SAAS service like SalesForce.com.

    Because of this, I expect that Intuit provide details actions plans that address this issue including provide all customers as summary of their audited Disaster Plan, Business Continuity Plan and Results from the last test including how we will be compensated for the lost revenue and expenses we incurred.

    Edward T. Bianco
    Chief Information Officer

  • Roberto

    Unfortunately, you did not explain why off-site systems were not able to handle the outage. You say that you know “more” but you haven't shared what “more” you know. This is too vague. Why don't you schedule a conference call with your top staff present and let us dial in and ask questions in real time? Instead, you have put the burden on us to open a support case.

  • Surya

    This is a common approach to any system and any solution. However, with Quickbase, being a cloud and a saas tool, all you can backup is data. But the more important part is the “code” that your QB Developers would have done, spending so many hours on that.

    There is NO way we can back that up, other than copy the application on to another Quickbase application. So the backup does not complete the picture. Without the code, just the data is not complete.

    We should have a way to make a backup of the code on to our own hard-drives, even if it means that it may not run without Quickbase live again. This will help us “restore” our code back to Quickbase, in case they lose our applications and the data.

  • Larry

    Please explain the power outage: was it utility power that failed? If so, surely Inuit has its own backup power system, whether a primary or secondary site? Or was there some meltdown in premises electrical wiring?

    And in any case, what is the prevention plan for future incidents? (My own wholly unscientific rule of thumb is that there's a 60% probability of recurrence of any single fault, no matter how rare…)

  • Howard Eisen

    Your comments, while appreciated, leave me (and all clients) wondering. What you describe is EXACTLY the kind of event that redundant disaster recovery measures are designed to prevent. This underscores an apparent lack of redundancy and back-up measures which is fundamental and essential to system outsourcing. Please let us know how that is to be cured. Immediately.

  • Warren

    Dear Ms Lee,

    I don't normally make comments on blogs like this but this time I feel I need to because I am a Quickbase developer and this outage impacted all my customers. The issue is not that my customers were unable to gain access to their data for a day (for the most part they can survive this type of situation), it really has to do with the fact that I have always sold them on the idea that the Quickbase systems had redundancy where in the event of a failure of one server, another would immediately take over. This is especially critical in the event of a natural disaster that takes out one of your server centers for an indefinite period rather than just one day (flood, fire, tornado, hurricane, earthquake, snow storm…). What I was always told was that Intuit ran redundant servers that were located in different geographic areas, which of course dramatically decreased the potential of both servers being impacted by a single event. Unfortunately, what you are now suggesting is that though you do run redundant servers, you have them located in the same facility. I am honestly quite surprised by this and now concerned that my customers may lose faith in a PaaS system rather than their own internal servers (which of course I realize can crash much more often than Intuit's servers, but it becomes an emotional issue rather than and logical one). As developers on “cloud platforms” we are constantly battling in-house IT staff that throw up concerns about the customer's data not residing on their own servers, and all the potential issues that may cause (mainly security and accessibility).

    This interruption has the potential of significantly diminishing customer confidence in Quickbase, and as a result I feel I need to ask if you could at least give us a better understanding of Intuit's server architecture, and more importantly your disaster recovery plans.

    Thank you

  • Paul

    Specifically, what accidental event caused the failure of your primary power. And second, what accidental event (specifically) caused the failure of your secondary power source. Third, what specifically are you doing to ensure this doesn't happen again. Speaking as one of your valued customers, I am not interested in hearing more responses from your public relations play book like those that have been read out loud so far. One of your other customers said it very well, trust in Intuit is your most valuable asset. You have lost a considerable amount trust with your recent outage, you better earn it back as fast as you can. Tick, Tock. We won't wait forever.

  • Donpital

    Intuit CIO, CEO, CFO, GET A CLUE!!! Haven't you learned anything from how to respond from BP and the Gulf oil problem??? Users (and in this case savvy technical ones) want and deserve good technical answers, not smoke-screen puff pieces written as if they've been vetted by attorneys:

    1. You have sold customers on redundant server locations in different geographic areas. You will satisfy me and probably eliminate a lot more poor publicity if you address the elephant or two in the room:

    A. Why were primary and secondary servers both in San Diego and why did both go down at same time/on same power grid?
    B. Where is/was the redundancy in the plan and did that fail also?
    C. Why didn't a different Data Center take over in a more timely fashion as advertised?

    I can understand the slowness in bringing systems back up for data integrity, but the reasoning behind why the existing (and hopefully redundant) disaster recovery plan failed is the question you are dancing around.

  • Rebecca Catlett

    I think everyone should be asking THEMSELVES why did this happen as well? Obviously there is a lack of ownership on the client's side as well. Due diligence is the cornerstone to every project and vendor selection. Yes, Quickbase needs to do a new risk management assessment, but I am also becoming aware that perhaps I didn't ask the right questions of Quickbase when I was going through vendor selection. Its easier to look outward than inward.

  • Klaus

    Thanks for the note. We are getting closer to the open dialog we should have seen much sooner. Unfortunately, it is too little too late. As many people are commenting on within this blog, we require urgent information on the following issues:

    1) Why did you not have adequate business continuity plans and fail-over systems in place for your application?
    2) What are you going to do to establish adequate business continuity plans and fail-over systems, and when will these be in place?

    Many of your customers are relying on your services and applications for business critical processes. We need to know that our information and processes are in good hands. Currently, we are not convinced this is the case.

    Respond to our requirements for this information, please.

  • Max

    Ginny, I can understand why you do not provide comprehensive details — anyone looking to attack your system could use them against you.

    However, I think Edward Bianco expressed it very well. To think that your production and backup systems are in the same geographic area, and in fact have the same power supply, is both astonishing and quite different from what Intuit salespeople have been saying for years.

    I am a consultant and I sell QuickBase and other SAAS products. I am up front about each product's plusses and minuses, and QuickBase is having a harder time stacking up.

    You have lost a lot of trust and right now I think the only way to get it back is to not just send us nice emails, but let us know that you are going to hold a DR test in which your San Diego system will be completely taken down — then we can observe what happens.

  • Allan

    The half hourly (or later hourly) updates on the website said nothing – sure, they were updated regularly (the time on the note changed, even if the wording didn't), but were about as useful as a chocolate teapot! I received absolutely NO meaningful updates during the downtime – not a single thing. I bet I'm not alone.

  • Allan

    Hang on! It's not crazy to expect the recovery procedures to act as advertised, as sold and as paid for. This clearly FAILED in this instance. Whether or not communication was good in a previous outage, it most certainly was not good in this instance. Frankly, it still isn't.

  • Eric

    I agree, we were left somewhat in the dark as to their progress and it would have been nice to know a rough ETA for everything to be back up. However, Allan, I'm also curious as to what you may have done with the information had Intuit provided more detail.

    While I don't feel that Intuit had the steps in place to prevent this and I'm still blown away that both servers were in the same location, I do feel they were doing everything the could to get things up as soon as humanly possible.

    I expect they're going to be taking a long hard look at their DR plan, and I look forward to hearing more about what changes are being put in place to ensure that this doesn't happen again.

  • TonyMontana

    This is the nuttiest post I've read. First of all, this “work-model” IS commonplace. I've been using Netsuite and Saleforce.com (the leaders in the SaaS model) since 1999/2000 and I've never seen anything like this. QuickBase is a wonderful product and we are developers and use it extensively internally, but what you fail to understand is that hundreds of thousands, maybe millions of businesses now rely on SaaS applications to run their businesses. They do so because of the advertised security and redundancy claims. It is crystal clear that the advertised redundancy claims were not in place.

    Crazies? How about customers and partners who expect Intuit, a global leader in SaaS applications, to meet minimum SaaS standards.

  • Danny

    You're out of service again now; this Saturday? Please stabilizize! -Danny

  • Klaus

    Hello Ms. Lee,
    When can we expect to get responses to the two major concerns we have raised in the comments in this blog?
    1) Why did you not have adequate business continuity plans and fail-over systems in place for your application?
    2) What are you going to do to establish adequate business continuity plans and fail-over systems, and when will these be in place?

  • Rod

    I agree with what Edward Bianco stated above. I would like a detailed plan of your disaster plan as well. Reason is that we no longer have the confidence in the “Intuit” name as we did when we first signed up for Quickbase.

    We have several clients relying on our customized SAAS applications as well and we need to make a decision if keeping Quickbase is the best option. The price we pay for Quickbase should give us peace of mind that Intuit is doing something to completely eliminate these types of power failures or outages. An hour or two is acceptable. A full day is definitely unacceptable.

  • David

    The outage was a significant disappointed, but the explanation for it by Ms. Lee was anemic in a manner reminiscent of Tony Hayward of BP. Ms. Lee made it sound like we would have all liked this to have worked out better, rather than taking full responsibility for the gap and providing detailed preventive measures. Our company has nearly 200 global users of QuickBase in the Americas, Europe and Asia, and a number of those users have expressed frustration.

  • http://quickbase.intuit.com/ Intuit QuickBase

    @David, we'd like to speak with you about the impact the outage has had on your team and talk about the business continuity options available for your users. We'll be reaching out to you shortly to discuss this further.

  • Rick Julian

    Apologies are nice but not sufficient in this case. Our business is very tightly integrated with Quickbase – it forms (formed) the backbone of our billing, our lead generation, autoresponses, etc. We lost over a full day of business as well as appearing incompetent to customers and employees alike. There is no way a multi-billion dollar tech company can't have adequate data centers.

    This is a very clear case of fraudulent advertising in promoting Quickbase and Intuit's other online products. Say goodbye to your cash reserves.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Hi Klaus, Thank you for taking the time to comment to this blog post. We will be reaching out to you directly in order to personally address your questions.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Max, we’d be happy to discuss what we have in place to protect your business. We’ve spoken with dozens of customers over the past several days. The email address you used here is not appearing in our CRM system, so please feel free to call us at 781-370-4414.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Hi Don, we'd love to set up a time to answer your questions. The following number 781-370-4414 will be good until July 1st so we can continue the dialogue.

  • Steve Leeke

    Astonishingly naive and amatuerish. Very disappointing.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Patti,

    The QuickBase move to the new data center is still planned as scheduled. We just sent an email out to Account Admins and Account Managers with the updated info.

  • candidcio

    This has happened to me on two occasions, despite separate power feeds. In fact one occasion the cause of the power failure was the back-up system. It is preventable, but not unfathomable.

  • candidcio

    I am not having the same reaction as most. My primary reaction is there but for the grace of God…

    Still, I believe you should be completing a Root Cause Analysis and sharing a summary of that publicly. That summary should address the engineering changes being put in place.

    I think you you articulated the communication problem well. You have to be clear with your users. At the point that you knew you had a hardware failure you should be able to predict a time window for the outage.

    The most sophisticated services in the world have outages: BlackBerry, Microsoft Danger, gMail. Certainly the 14,000 users I serve have suffered through outages (although it has been a long while since we have had one of this magnitude). I do not have an expectation of 0 downtime. The key now is to learn from it so this is doesn't happen again.

  • candidcio

    This was posted by an employee on our Yammer:

    10) Archive and purge your email inbox
    9) Schedule all those meetings
    8) Work on that Status Report
    7) Make a few phone calls
    6) Go into the Timekeeping application to record your planned PTO in November & December
    5) Re-read the Project Charter and see what Deliverables you've been ignoring
    4) Engage in a marathon strategic planning session
    3) Read up on the Help Files for some application you want to learn more about
    2) Investigate Zoho Creator as a software alternative

    And finally…

    1) Draft a recommended Disaster Failover Plan and send it to Intuit

  • gb

    I think this is something all customers would like to know. Why would you not post this publically or send out infomation to ALL customers since we ALL were impacted? I don't think it is fair to ask only irrate customers to take time to contact you for restitution or resolution. I am concerned now about the scheduled outage this weekend and the assurance that we will not suffer additional down time going forward….

  • Jesus

    Klaus – what was their explanation when they reached out to you?

  • Databrain

    DR & BC has and will continue to be a specialized area of I.T. for over 30 years. Having worked at AT&T in the Corp. IT Data Centers, there is no excuse for not having a tested, tried, and true DR/BC plan. However, in many companies this critical area of IT is often treated as a stepchild.

    The reality is – a lot of papers are written, a lot of “procedures” are identified, and a lot of plans are documented. Typically, there is not a lot of is money to walk the talk. I suspect this is the case here.

    Yes, Intuit has lots of data centers – as we've been told. However, the money to implement true DR/BC has not been invested to create the platform(s) needed to actually support near-real-time recovery.

    Thus, IMHO the real blame falls on the Executive Board for not addressing this real concern with the support required to deliver on “meeting and/or exceeding customer expectations.” Now that they've been “caught” – the risk exposed – tis up to them to fix it.

    The time for excuses is over – the time for action is now.

    Tim Whitney
    Chief Technology Advisor

  • Concerned Customer

    I'm witnessing your destruction of your brand.

    You unfortunately exposed your flaws during an unacceptable outage – in customer communications and in systems architecture. Now you follow that with:
    1) A PLANNED OUTAGE on it's heels (unacceptable)
    2) A outage window that is outrageous (8 hours planned downtime – unacceptable)
    3) A planned TIME for that outage that appears convenient for your staff (930am-530pm PST), certainly not for your customers. (unacceptable)

    The arrogance and distance from your customers this displays is so destructive to all you have built. I can't help but think of Toyota, but thus far, it's worse – you display no awareness of how this is impacting your reputation and BRAND. This is trashing your brand – trust me. From one Board room in fortune 100 – QB is a platform to move away from. Brand value is precious, SOOOO difficult to earn, it's so hard for me to watch you burn it an instant.

    Please reconsider your data center move plans NOW.

  • http://quickbase.intuit.com/ Intuit QuickBase

    @Concerned Customer – Phil just answered these exact questions in QuickBase community. http://community.intuit.com/posts/qa-about-the-

  • Allan

    I couldn't agree more. And as someone who has been “reached out to” yesterday, I'm afraid the information we need is still not being made available – the guy who called me did not have answers to most the questions and concerns people are raising on these comments – he promised to email me back yesterday too, and still hasn't done so. Communications from Quickbase seem not to be getting any better. It's a real concern.

  • Allan

    To compound everything else which has happened with the outage, NOW QuickBase are saying that their communication to users about this weekend's scheduled maintenance gave the WRONG times for downtime. So having expended significant resource in briefing users for the downtime, now we have to begin the whole process with workers on an earlier shift with less than 24 hours notice. Nice one, QuickBase!

    So to other users… Beware – it's NOT Pacific Time as communicated in their original email about the downtime, the downtime begins this Saturday at 9.30am Eastern Time.

  • Allan

    I totally agree.

    But beware – even the times they told us all that the downtime would take place are wrong! It's ET, not PT. They didn't think to email everyone to tell them about the mistake so customers can plan around it – when I became aware of this latest stuff-up, I suggested they get an email out urgently to warn their customers, but so far there's no sign of that. So if you made plans based on PT, you'll get a 3 hour shock tomorrow – effectively 3 more hours of unexpected downtime.

    Unacceptable is definitely the word.

  • Aclawson

    Service was up/down most of yesterday and is down again today. Why can't you get this right? After all of these years without any significant problems what changed? Who was hired/promoted and started to play with the system?

  • Ron Bovick

    Now what is the problem? Quickbase is starting to scare me with it's unreliability. Is our data safe and secure?

  • TonyMontana

    You guys are killing us. QuickBooks has either been down or completely unreliable over the last couple of days. Not only do we, and our clients, rely on QuickBase to run our businesses, but we're also relying on QuickBase to be functional so we can demo/sell it and build applications for our clients.

    When are we going to have some kind of a roadmap so we know what's going on and when these issues will be fixed?

  • Allan Hawkey

    I thought yesterday's email about the slow and intermittent service had been a step in the right direction – it told us about the problem and what was being done about it.

    However, yet again, expectations have been badly set. According to the email, there was going to be downtime of about 30 minutes between 8-10pm PT – for our business, in the UK, this would have been through the night, so not a problem.

    Now we find that the business day here has begun and we are left without service again. Worse, the service page tells us there won't even be another update for 2 hours, so I assume service is likely to be down for at least that long.

    Who at QuickBase is responsible for the latest miscommunication in yesterday's email? Who is responsible for the serious maintenance overrun? And who is responsible for the ludicrous decision not to update the service page for 3 hours or even give a realistic expectation of when service will resume?

  • Ron Bovick

    Dear Brad, Ginny and Rhonda- When our monthly service fee is due to you this month, I'll be sure to give you updates on the delay of payment, as my time will be used on creating excuses for my clients on why I cannot deliver my service to them and access their valuable data on your service. They in turn will be laughing at me when in turn I submit my invoices to them on 8/1, furthering the delay of payment.

    And oh yeah, I changed banks last month (like you changed data centers), but the branch manager lost the keys and I can't get in to withdraw.

    Sorry, but I'll you keep updated on progress

  • PeteB

    After over 2½ years with QuickBase, we had been very pleased with the reliability. Now we're seeing the third severe outage in just over a month.

    What's more worrying, is that having believed we were dealing with a company which had world-class backup and recovery arrangements, we’re seeing little evidence to support that belief. On top of that, after a couple of days of contrite apologies over a month ago, and promises of more information, everything had been quiet since and we still don’t know what happened. Neither have questions about replication in two separate data centres been answered.

    It’s the lack of information and answers that’s at least as likely to drive us to a different solution as the outages themselves!

  • Allan Hawkey

    Getting information from QuickBase about the 25 hour outage last month was very difficult. Eventually, if you're willing to persevere and talk to someone senior enough (at Intuit, rather than QuickBase), you do eventually get some sense from them. Actually, they were quite good in the end.

    Though, having said that, you're absolutely right that the information QuickBase promised to send out about the last outage has never materialised. Perhaps they think we've forgotten about it? Or maybe their investigation is still ongoing (though what they could possibly still be investigating this long after the event is beyond me)?

    I was also told that the new data center would make it virtually impossible for a prolonged interruption to happen again. Yet here we are… Another outage that seems to have taken down BOTH the primary AND secondary systems in the (new) primary data center, and there is apparently still no easy mechanism for switching to the secondary data center as it takes too long and results in data loss (which is what they told me last time). Seems to me that there is still no effective plan for service continuity in place.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Tony,

    We are posting all the current information we have to our service page. The best thing to do is to email our support team at supportrequest@quickbase.com so we can address your case individually.

    We understand how critical the proper functioning of QuickBase is to your business and are working around the clock to fix this problem.

    — The QuickBase Team

  • http://quickbase.intuit.com/ Intuit QuickBase

    Allan,

    We cannot give you an ETA for when service will resume. As you may have read, we have completed installation of a patch to address the connectivity issue between the QuickBase application layer and our back-end storage infrastructure. Following final validation testing, we will restore the QuickBase service. When we have more information on our progress it will posted to our service page.

    Please email our support team at supportrequest@quickbase.com so we can address your case individually.

    — The QuickBase Team

  • http://quickbase.intuit.com/ Intuit QuickBase

    Pete,

    As soon as we have information on our progress, we are reporting it on the service page. As you may have already seen, we have completed installation of a patch to address the connectivity issue between the QuickBase application layer and our back-end storage infrastructure. Following final validation testing, we will restore the QuickBase service but cannot give you an estimated time of when the restore will be complete.

    Please email our support team at supportrequest@quickbase.com so we can address your case individually.

    — The QuickBase Team

  • Allan Hawkey

    Without an ETA, what are your suggestions for how we plan and run our business today. Do we keep staff in or send them home?

    Regarding email your support team, I did this 3 hours ago, and have not received any response. What's the point of emailing if noone replies?

    And I note that you've overlooked my questions about who is responsible for these latest failings. I'm sure, in the spirit of openness which was expressed by QuickBase after the last outage, that you won't want to hide this information from your customers.

  • Allan Hawkey

    Tony, maybe you'll succeed in getting a reply – I emailed them hours ago, and haven't had any reply. Good luck!

  • Eric B.

    The point of Tony's post is that your support updates are generic at best. Almost a month to the day since the previous major outage, we need some real assurances that the root causes are identified and being addressed. We've been clients for 5 years, why all of a sudden these problems?

    This is a business application, and I assume most of the people viewing this forum at this hour are doing so because we are responsible and accountable to answering this same question to clients, co-workers and managers very soon this morning:

    Why all of the problems recently, and what is being done to prevent this from happening in the future?

  • C.Catrambone

    Dear Quickbase, Our small company too is in complete shutdown at the moment. This has become routine during the past several months. We rely on QB to manage our global operations our business is suffering. The current shutdown has cost us a serious amount of money and has questioned Quickbase's reliability. Once is a year is fine, but three incidents that I can recall this year has seriously hurt your reputation in our company. I would like to see Quickbase credit our montly subscription fees, since this is not what we signed up for. We pay for your advertised worldclass backup. It is all very frustrating and still, our operations are down as I write this.

  • Ken Ingraham

    On that note I was shocked to see QB continued charging our account after last month's unacceptable outage.

    In my business, microscopic in relation to Intuit, we make plenty of mistakes too. our policy is to give triple the amount of damage we have caused. Once to cover the damage, twice as compensation, and a third time as punishment to ourselves. Will be interesting to see how Intuit compensates its loyal customers for its shameless acts of fraud.

  • PeteB

    Thank you, but this is beyond a support case.

    I would like, demand even, a response from senior management that answer all the questions raised over a month ago! – which they promised to do, and I certainly don't think I should be spending my time chasing Intuit management for answers myself – their customers deserve better.

    PeteB

  • http://quickbase.intuit.com/ Intuit QuickBase

    C. Catrambone,

    We understand how critical this outage is for your business. Please send an email to our support team so we can open a case and address your issues on an individual basis: supportrequest@quickbase.com

    — The QuickBase Team

  • Roy Graves

    For a little more than I pay for Quickbase each month I bet I can find a hungry DB developer willing to take monthly payments for them to develop a fully customized application to replace the one I currently have on QuickBase. That extra amount each month will be worth it to have happy customers and a happy owner.

  • Concerned

    Translation:

    “These interns just aren’t working out. No matter how many Hot Pockets and 2-liters of Mountain Dew we throw at them, they are seemingly unable to cobble this piece of horse dung into something we can pass as workable. We’ve been calling all of our vendors looking for a solution, and the stuff they sold us is even worse than the shit we bought in the first place, so we don’t know who to blame.

    We believe that it was an unwise move to outsource our hosting to “Dreadlord1337” who, apparently runs this business out of his mom’s basement using cold-war era servers that take a dump every time a poodle farts. As soon as we are able to get your data out of Dreadlord1337’s “data center” we’ll have this pile of manure up and running. Could be today, could be never.

    Thank you for your patience,

    The Quickbase team.”

  • Jessica White

    So far, the response from Quickbase is awful. Here's how it could be improved.

    When faced with a situation like this, Quickbase should:
    1. Apologize profusely, repeat the apology incessantly, and mean it. So far, their responses seem rote and defensive to me. Not once, in any of their explanatory memos, have they said “I'm sorry” or “We're so sorry about the trouble we've caused”. Sorry goes a long way. Not euphemisms for sorry, but just “I'm sorry”. End of sentence.
    2. Be much more transparent about the technical problems they are experiencing. Quickbase has a high proportion of technical users, and vague updates are insulting. Be direct with your users about the exact cause of the problem. Help us understand what takes 14-30 hours to fix.
    3. Improve the infrastructure. There is no excuse for having all functionality housed in ONE data center, no matter how new and advanced it is. Quickbase should be hosted at multiple, redundant sites so that if one has problems, the others take over.

  • ahawkey

    Jessica, you make very good points. Just take a look back at the comments from the last major outage a month ago – the really sad thing is that the mistakes this time are just the same as last time. Insincere apologies, poor communications, and fundamental flaws in business continuity planning and technical infrastructure. None of the lessons of last time seem to have been learned.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Jessica,

    We appreciate your feedback and fully understand that our customers are highly technical. Please voice your concerns, frustrations, comments in our Community forum (http://community.intuit.com/category/quickbase) and if you haven't already done so, email our support team at supportrequest@quickbase.com so we can address your issue on an individual basis.

    — The QuickBase Team

  • Very Distraught

    “We will update this page at 8:00am PT today or as soon as we have more information”…….. It's 8:19 already, still no further update… c'mon now…….

  • Roy Graves

    Your recovery prediction at 5am of 2-3 hours was off by 6 hours. Really?! Can you tell us one more time how much you understand how important QuickBase is to our business and how we should email support so our concerns can be addressed?

    Our concerns are that your system is down, unavailable and unreliable. It can be addressed by fixing these problems, now. Get a clue.

  • John Maynard

    The 8am PT service update states the system will likely be down until 2pm PT (or 5pm my time). There goes another day…unbelieveable. And please, don't hit the send button for your canned response directing me to contact customer service.

  • Steve

    Fool me once shame on you, fool me twice, I'm taking my business to TrackVia.

  • Steve

    And did I mention you guys are full of SH*T! First a power failure, now a data leakage??? Are you kidding me? You serve Fortune 100 companies, you really think we're all idiots? Just insulting. I'm positive that none of us customers would want to know the real reason for all the trouble as of late, which is more than likely they have been subject to some large scale cyber attacks as they would be a huge target for their status in the SAAS industry as well as for the valuable financial data held in their online Quickbooks software.

  • Steve

    On an individual basis?? That means you're not really willing to pony up and offer all your customers what they deserve – just the ones that make a lot of noise. After the magnitude of problems you have caused in the last month your response ACROSS THE BOARD should be that of an product recall. Offering a solution to ALL customers, not just the ones who “voice concern”.

  • Steve

    Wow deleting comments, I hope you realize I have ALL F-ING DAY! Being that I can't work and all.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Steve,

    We have not taken any comments down. We are actively listening to all comments and responding where we can and its appropriate.

    Our main channels for communication are the service page, our Community forum (http://community.intuit.com/category/quickbase), and our support team (supportrequest@quickbase.com or +1 (781) 370-4414.

    — The QuickBase Team

  • Na

    There isn’t much to add, but with so many outages in the last 45 days, this becomes plainly unacceptable.

    We can’t rely on Quickbase any longer to be a Business Ready application.

  • QBuser

    I would second that, “a response from senior management” is a must!

  • Aclawson

    If all you need is the raw data then you are spending far too much money on what could easily be handled by an excel spreadsheet.

    Quickbase is in the middle of yet another 24 hour (or more) downtime. There is clearly no redundancy in their systems and it really looks like people are putting things into production before they have been load tested. System-crippling “memory leaks” do not just appear in the production environments unless somebody changes the system, and nobody should change the system unless they have tested in the sandbox first.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Comments and concerns are best voiced on our Community forum. Feel free to start a thread: http://community.intuit.com/category/quickbase

  • http://quickbase.intuit.com/ Intuit QuickBase

    Comments and concerns are best voiced on our Community forum. Feel free to start a thread: http://community.intuit.com/category/quickbase

  • Jessica White

    Intuit: Do you want us to express our frustration in the Community Forum because you have to have a login ID to read it? And this blog page is public? Because it seem that many of us are quite content to vent our frustration right here. Speaking only for myself, I've found it very interesting to listen to some of the very constructive and pertinent criticisms that have been expressed on this blog, and since the frustration goes back to the June outage (even if the causes of the outages are different), it somehow seems appropriate to hold the discussion here.

  • Very Distraught

    So the countdown is on to the next update… where we find out it'll be another 6 hours. Then another…. then another….. wow…….

  • G Reneau

    I have 15 employees sitting around unable to work and several dozen other external users seeking updates and getting nothing on their files. I am in the real estate business and people are depending on us to save their homes. The outages are unexplained and unacceptable. You have cost my company thousands of dollars in lost wages and production. If I didn't have a custom built app and all of my data houses in your system I would have made the switch already. I am so angry I can barely type this.

  • O Victory

    I use QuickBase as our entire Work Force Mangagement. We have 118 users who have been down. All of our jobs and invoicing are done through QuickBase, which we now have no access to. Will it be up and running tomorrow morning 9am eastern time.

  • http://quickbase.intuit.com/ Intuit QuickBase

    O Victory,

    Unfortunately, we do not have an ETA for restoring QuickBase to service, but we're continuing to make progress. We still don't expect to be up any earlier than 4:00pm PT today. Any service update will be posted to the Service Page: http://service.quickbase.com/updates.aspx

    Please email supportrequest@quickbase.com or call +1 (781) 370-4414 so that we're aware of your specific concerns and can address them accordingly. If you already contacted us, we are working around the clock to get in touch with all customers as soon as possible.

    — The QuickBase Team

  • O Victory

    Your killing me with the canned response. I would call or email (the links above) if I thought I would get a answer.

  • Ron Bovick

    This made me laugh. I've been thinking the same thing, but could not articulate in such a humorous fashion. Cheers

  • Roy Graves

    Intuit,
    That is it for me. This has finally pushed me to develop my own application to replace my QuickBase application. It will not happen tomorrow or next week but you can be sure that you have lost me as a customer. Your updates page showed 95.76% availability for the last 90 days. That is just not good enough to meet expectations. Like so many of your other customers, my applications are CRITICAL to the operations of my company and having them unexpectently unavailable for 2 days is unacceptable.

    The philosophy of my company is along the lines of do everything possible for the customer and the profits will follow. Based on my experience with Quicken, QuickBooks and QuickBase, I believe the philosophy of Intuit is make a profit right away by doing just enough to be slightly better than our competitors. It is a gamble for both of us and in this case, you loose. How sad.

  • G Reneau

    In an update to my earlier post, my application is back up, and apparently missing all new records created in the last 2 months+. This is much, much worse than the database experiencing an outage. This could literally cripple my company if not resolved quickly. I am so exhausted by this process. I fail to understand how this could happen repeatedly. A service interruption is one thing; a loss of critical customer data is inexcusable.

  • Jessica White

    Oh Geez, that's really bad. I hope they are able to recover your data and FAST! How horrifying. Have they responded to you individually yet? Keep us updated–I'll be interested to hear what happens.

  • http://quickbase.intuit.com/ Intuit QuickBase

    Glen,

    If you haven't already done so, please submit a support case here: https://quickbase.intuit.com/support/portal/#YW

    I am contacting you directly through your email above to follow up on this post.

    Alexandra Hastings

  • Chris

    Did the site just go down again?

  • Tlilly

    Quickbase down again??
    Are you kidding me????

  • Dugie

    How can a known issue that prohibits Quickbooks online customers from connecting with their bank (Citizens Bank) be permitted to continue for 10 days? Also, why is it that customer care knows about the issue but can’t trouble shoot it? While I’m on the topic, why can’t Intuit properly download transactions from a global vendor like Citibank properly?

    Where are you folks? It’s time to look out of the ivory tower and understand customers’ issues.

  • Bill

    As CIO you should know that your company has just lost ALL of my emails and claims they do not have the ability to restore. Stop selling email as a product if you can’t restore.

    • Alex Hastings

      Bill — Someone from QuickBase Support is reaching out to you ASAP.