We’ve got data working it’s way back into ALL accounts, which you’ll see today.  We’re in the process of fixing the CSV exports and getting everything else working as it should.  You may still some keywords missing data, but those should be restored soon as well.

Since this has been such a big outage, we’ll be moving the billing date back 10 days on all accounts, crediting everyone 10 days worth of data.  This will also give any of you who’ve signed up in the last week, 10 extra days to let us earn your business and make up for the lack of consistent service.

Keep checking the blog for updates … we’ll have everything going again soon.

Over the last week we’ve been trying to figure out exactly what’s causing all these issues.  Wednesday night we were finally able to isolate the problem to what we thought were only two bottlenecks. We optimized those that night and they were operating as expected.  We woke up this morning to realize that there was in fact a third problem which needed to be resolved.  We’ve been troubleshooting throughout the day, and I’m glad to say that the last pain point to getting data into everyone’s accounts is resolved, but since that process is the final entry point from data collection to the interface, it’s going to take a while to get caught up again.  We’re processing search results, which were aggregated Wednesday.  Those should be finished in a few hours, but then we still need to process the search results which we gathered yesterday and do some calculations on those to get everyone’s accounts looking normal.  So … by this afternoon we should be caught up and have Wednesday’s and yesterday’s data available.

Again, thanks for your patience. We realize how extremely unacceptable this is, and are taking measures to prevent anything remotely as drastic from happening in the future.

We really appreciate everyone’s patience as we get this handled.  Pretty sure we’ve got everything figured out and everyone’s accounts should have good data tomorrow.  There may have been some keywords which slipped through the cracks today, but those should get picked up and back to normal the next day. @joseuonline pointed out that the end of the month is very soon, and we’ll for sure have good data by then, so everyone can get all those reports prepped and sent out next week.

We’ve been working day and night over the weekend to get everything moved over to Amazon smoothly. It didn’t go as smoothly as we had hoped though. Yesterday we had problems that could be resolved by today, so there are going to be issues with your account.

Thanks for your patience as we work through this.

Update: Things are back to normal now.

Early this morning the process that pulls yesterday’s ranking data into the interface was delayed about five hours. Your account may look wonky, and be really slow, until this completes. It should be soon, but I just wanted to let everyone know what was going on. We do have your data, it just may not look like it at the moment. Everything will be back to normal in a few hours. Thanks for your patience.

Since we’re talking backend things at the moment … might as well update everyone on what we’re working on. For the last few weeks we’ve been updating the entire architecture to be much more reliable, flexible and scalable. Part of this process is getting everything on Amazon’s cloud. We’ve been on Slicehost up until now, but are running into some issues with them, as you may have noticed today Amazon will not only allow us to scale the way we aggregate and process your data, but they also provide us with quite a few utilities to take advantage of in the process.

Again … thanks for your patience as we make this transition. Things will be a LOT better here shortly!