Page 1 of 1

Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:13 pm
by AUbicycles
Hello,
I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.

This means that posts have been lost (weekend) and I apologise for this lost and hope that you can forgive this - both the host and myself have worked to find the quickest most acceptable solution to getting this back online. It also means that brand new members who have registered will need to register again.

Comments are welcome... and I am currently looking to get the main site online as well.

Cheers
Christopher

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:16 pm
by The Womble
Ooh yeah! im the first poster yeah!

You can still access most threads from the Weekend Chris

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:35 pm
by peter
I thought it's Groundhog Day!

By the way, just got SQL error again.

Hope the host can track down the cause and provide a fix.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:37 pm
by twizzle
peter wrote:I thought it's Groundhog Day!

By the way, just got SQL error again.
+1 to more SQL errors, and I'm missing PM's from the middle of last week. :(

Edit: O.K., it was probably from late Friday that I lost messages. My memory is sh!te.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:37 pm
by CommuRider
Oh well, at least I managed to read the Gong posts....

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 9:56 pm
by schroeds
(sob) my hairy ride thread has disappeared and it was shaping up nicely :cry:

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 08, 2010 11:05 pm
by queequeg
AUbicycles wrote:
I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.
This is when DBAs can justify their salary :-) I've had my fair share of these in the last 15 years, but unfortunately the data has been a bit more "critical" than forum posts. I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL :-)

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 6:26 am
by Max
queequeg wrote:I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL :-)
What the! You stay the hell away from our forums! Who knows just how much of a hand your "influence" had on this event!! ;)

Seriously - thankyou Chris for all your efforts in getting the site back up and running. We lost some posts, so be it. Better than losing the entire board.

:)

Max

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 7:00 am
by Comedian
Having been there too I appreciate your efforts!

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 7:01 am
by Apple
Yes, I had a PM waiting for me to read, it could have been great gossip and who ever sent it please send it again. :D

Good job on getting it fixed, I am sure it wose a son of b%^ch to fix.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 9:39 am
by AUbicycles
Thanks everyone.
My contact with websitemanagers was also travelling interstate and so took time on the fly to help support and resolve. Credit.
And... we have been doing very very well, no issues for a while.

Still interested if we can get to the bottom of this and see what preventative methods we can get into place.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 7:04 pm
by Saro28
top job blokes, good luck finding and fixing the prob' (cue Thumbs up emoticon here)

Andy

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 11:29 pm
by snark
AUbicycles wrote:Still interested if we can get to the bottom of this and see what preventative methods we can get into place.
For the preventative part, how big is a dump of the database (compressed or zipped)? Maybe this can be backed up multiple times a day, and if it compresses well you could keep a reasonable number of backups, perhaps off-site.

Probably stuff you've thought about already, but maybe worth a mention...

Cheers,
Simon.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 09, 2010 11:51 pm
by AUbicycles
The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Wed Nov 10, 2010 6:08 am
by dclnmurray
I do a bit of database admin work so know the hassles involved. If the client doesn't use any database replication (database mirroring etc) our standard is using maintenance plans to do a database backup in the early hours of the morn and transactional hourly backups in between. Assuming the DB is not in simple mode. This allows recovery model to the most recent transaction backup although I am not aware of how MySQL works in this regards.

Anyway, appreciate the work done.

Dave

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Wed Nov 10, 2010 10:16 am
by AUbicycles
Thanks Dave... good thoughts.
One thing that caught us out were the MySQL errors and not being able to pinpoint exactly when the errors may have crept in.

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Wed Nov 10, 2010 10:24 pm
by queequeg
AUbicycles wrote:The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.
So it is quite a small database then ;-)

I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!

I don't have any experience with MySQL, but most of the databases I deal with these days are sized in Terabytes. Our backup strategy is a mixture of technologies, depending on the specific requirements. The most critical system all run standby databases at secondary sites with "near real time" syncing of transactions. of course, if you get a logical corruption in the transaction logs then it does not help. In those cases it is restore from backup and roll-forward to the last transaction before the corruption.

Personally, you have not lived until you have been working on a Severity 1 Database Corruption issue with the Database Vendor for 72 hours straight, trying to isolate the corrupt blocks and somehow bypass them so that the database instance stops crashing. Life as a DBA is never boring :-)

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Fri Nov 12, 2010 7:47 pm
by twizzle
queequeg wrote:I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!
I did performance reporting on a system that had ~ 1 billion rows in the main table, despite weekly culling of old data. Mainframe VSAM... then they tried running it on DB2 instead. Didn't perform anymore. :roll:

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Fri Nov 12, 2010 7:51 pm
by CommuRider
Had the SQL error around 5pm. Is it stable again or is this a brief respite?

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Fri Nov 12, 2010 7:54 pm
by The Womble
Those warrrr the days. I 'member havin't eat a handfull 'coal, trudge down t' mineshaft t' office, work with 14 billion lines of data whilst sittin' on t' chair covered in broken glass...

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Sat Nov 13, 2010 10:07 am
by AUbicycles
Updates are going here. This issue was related to the weekend problem as a (failed) attempt to improve performance. Host was onto it quickly and more errors are not expected.

Cheers
Christopher

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Mon Nov 15, 2010 9:20 am
by Myddraal
A handy site for next time; howf***edismydatabase.com

Re: Server Error - Forum Restored from Backup - Posts Lost

Posted: Tue Nov 16, 2010 9:36 am
by AUbicycles
:)