Database errors

As many of you noticed, there was a big database trouble going on yesterday.

The problem was that some innodb tables went corrupt.
The affected tables were some on another database, and one table in the wechall database, namely "wc_chall_solved", which holds the records of who solved which challenge.

Querying the corrupted tables caused a connection loss to the db, which occured quite a lot. In total i got like 30000 emails containing db error messages

Sadly i could not repair the InnoDB tables, so i had to replace the corrupted tables with a backup that was 36 hours old.

Long story short:
If you solved a challenge yesterday or the day before, your solved status might got lost.

Your percentage and scores were not affected, so you will not gain more points when resolving them.

I am sorry for any inconvinience, but it could have been worse

Happy Challenging!

PS: During this week i am really busy, but will have more time again from next week
So take my apologizes if i did not answer PMs or Forum Posts.

Quote from Gizmore

Feb 16, 2011 - 16:23:56

Long story short:
If you solved a challenge yesterday or the day before, your solved status might got lost.

This only effects WeChall challenges, right? It does not effect any scoring or solved percentages that may have come from a linked site?

Correct. Only the own wechall challenges were affected.
Scores and percentages have been untouched.

My lesson learned: if innodb fails, it fails hard.

Luckily there are daily backups

Quote from Gizmore

Feb 16, 2011 - 17:07:00

My lesson learned: if innodb fails, it fails hard.

Luckily there are daily backups

From a BCP perspective, perhaps it's a good idea to periodically perform a parallel test [1]. This (preventive) control makes you more proactive rather than (in this case) reactive. Easier said than done, I know

[1] http://www.riskythinking.com/glossary/parallel_test.php

Today i re-imported the database into a fresh db, and switched to MyISAM.
The dump seems complete, but It could be that some data is lost, so if you find something suspicous, please report it to me.
So far there were no more errors since i did the re-import.

I am sorry for the inconvinience over the last week