Thursday Evening Darktide Crash and Results

Original Link (now dead) - http://ac.turbinegames.com/index.php?page_id=163

Thursday Evening Darktide Crash and Results

---

Posted on 23-Jan-2004

What happened Thursday night? It appears that, during the second phase of the AC datacenter move, one of the Darktide servers was physically jostled in some way and began to fail. The server crashed twice and then we began seeing reports of lost items, particularly items lost in houses. The reports included whole settlements completely empty of items. Over 150 houses were affected. We restarted the world in an attempt to resolve the issue, but it did not.

We immediately began investigating why this occurred and what we could do about it. Our initial reaction was to restore Darktide to the previous backup. This would be a rollback of several hours, but all lost housing items would be restored.

Unfortunately, this is not an option. The AC datacenter move was still ongoing – the second phase was not complete at this point – and the previously moved servers had not been set up on the tape backup system at the new datacenter yet. There was a local drive backup... but it was made after the problem occurred.

The most recent Darktide backup was from immediately before phase one of the move started – approximately 8AM PST / 11AM EST Monday morning. In addition, as this backup was made at the old datacenter, we would not be able to migrate the data and restore it until next Monday, the 26th.

We had a dilemma on our hands – roll back Darktide to Monday morning or not? Rolling back would restore the housing items lost, but players would lose all XP and items that they had gained since Darktide came back up on Tuesday. If we did not roll back, the housing items would be lost forever.

We investigated other options, such as attempting to roll back only housing data or identifying those players affected and reimbursing them somehow. Unfortunately, we were not confident in our ability to do either of these things without the possibility of introducing more problems.

This was a very, very difficult decision to make. We had to decide which we felt was more important – those items which affected players may have kept in storage in their homes, or the items and XP that all players on Darktide may have gained during the time since the Monday backup.

In the end, we have made the following decision:

On the morning of Monday, January 26th, Darktide will be rolled back to the last good backup – last Monday’s.

During the weekend, we will be making periodic world broadcasts on Darktide to inform those players who may not be active on the web or on the forums of the rollback.

We will also be doing the following to reimburse Darktide players for their lost XP and items:

• When Darktide comes back up after the rollback, all in-game housing will have its maintenance paid for one rent cycle.
• In the February event, Darktide players will be able to use Town Criers for a one-time reward: two pieces of high-quality treasure-generated loot items as well as enough experience to gain three levels. Characters who are level 126 will receive three times the amount of experience required to go from level 125 to 126.

Making this decision was very difficult for us and we would have liked to have been able to resolve this in other ways. Unfortunately, we do not currently have the tools to allow us to do so. Once we have completed the full transition of Asheron’s Call from Microsoft to Turbine, we will begin working on tools that will allow us greater tracking and flexibility, should a worst-case scenario such as this one ever happen again.

To the players of Darktide: We apologize wholeheartedly for this incident. We wish there were a better option available to us but, given our options, we feel that this is the best course of action. Please accept our heartfelt apology.