Server Error - Forum Restored from Backup - Posts Lost

[AUbicycles]

Hello,
I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.

This means that posts have been lost (weekend) and I apologise for this lost and hope that you can forgive this - both the host and myself have worked to find the quickest most acceptable solution to getting this back online. It also means that brand new members who have registered will need to register again.

Comments are welcome... and I am currently looking to get the main site online as well.

Cheers
Christopher

[The Womble]

Ooh yeah! im the first poster yeah!

You can still access most threads from the Weekend Chris

[peter]

I thought it's Groundhog Day!

By the way, just got SQL error again.

Hope the host can track down the cause and provide a fix.

[twizzle]

I thought it's Groundhog Day!

By the way, just got SQL error again.

+1 to more SQL errors, and I'm missing PM's from the middle of last week.

Edit: O.K., it was probably from late Friday that I lost messages. My memory is sh!te.

[CommuRider]

Oh well, at least I managed to read the Gong posts....

[schroeds]

(sob) my hairy ride thread has disappeared and it was shaping up nicely

[queequeg]

I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.

This is when DBAs can justify their salary I've had my fair share of these in the last 15 years, but unfortunately the data has been a bit more "critical" than forum posts. I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL

[Max]

I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL

What the! You stay the hell away from our forums! Who knows just how much of a hand your "influence" had on this event!!

Seriously - thankyou Chris for all your efforts in getting the site back up and running. We lost some posts, so be it. Better than losing the entire board.



Max

[Comedian]

Having been there too I appreciate your efforts!

[Apple]

Yes, I had a PM waiting for me to read, it could have been great gossip and who ever sent it please send it again.

Good job on getting it fixed, I am sure it wose a son of b%^ch to fix.

[AUbicycles]

Thanks everyone.
My contact with websitemanagers was also travelling interstate and so took time on the fly to help support and resolve. Credit.
And... we have been doing very very well, no issues for a while.

Still interested if we can get to the bottom of this and see what preventative methods we can get into place.

[Saro28]

top job blokes, good luck finding and fixing the prob' (cue Thumbs up emoticon here)

Andy

[snark]

Still interested if we can get to the bottom of this and see what preventative methods we can get into place.

For the preventative part, how big is a dump of the database (compressed or zipped)? Maybe this can be backed up multiple times a day, and if it compresses well you could keep a reasonable number of backups, perhaps off-site.

Probably stuff you've thought about already, but maybe worth a mention...

Cheers,
Simon.

[AUbicycles]

The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.

[dclnmurray]

I do a bit of database admin work so know the hassles involved. If the client doesn't use any database replication (database mirroring etc) our standard is using maintenance plans to do a database backup in the early hours of the morn and transactional hourly backups in between. Assuming the DB is not in simple mode. This allows recovery model to the most recent transaction backup although I am not aware of how MySQL works in this regards.

Anyway, appreciate the work done.

Dave

[AUbicycles]

Thanks Dave... good thoughts.
One thing that caught us out were the MySQL errors and not being able to pinpoint exactly when the errors may have crept in.

[queequeg]

The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.

So it is quite a small database then

I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!

I don't have any experience with MySQL, but most of the databases I deal with these days are sized in Terabytes. Our backup strategy is a mixture of technologies, depending on the specific requirements. The most critical system all run standby databases at secondary sites with "near real time" syncing of transactions. of course, if you get a logical corruption in the transaction logs then it does not help. In those cases it is restore from backup and roll-forward to the last transaction before the corruption.

Personally, you have not lived until you have been working on a Severity 1 Database Corruption issue with the Database Vendor for 72 hours straight, trying to isolate the corrupt blocks and somehow bypass them so that the database instance stops crashing. Life as a DBA is never boring

[twizzle]

I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!

I did performance reporting on a system that had ~ 1 billion rows in the main table, despite weekly culling of old data. Mainframe VSAM... then they tried running it on DB2 instead. Didn't perform anymore.

[CommuRider]

Had the SQL error around 5pm. Is it stable again or is this a brief respite?

[The Womble]

Those warrrr the days. I 'member havin't eat a handfull 'coal, trudge down t' mineshaft t' office, work with 14 billion lines of data whilst sittin' on t' chair covered in broken glass...

[AUbicycles]

Updates are going here. This issue was related to the weekend problem as a (failed) attempt to improve performance. Host was onto it quickly and more errors are not expected.

Cheers
Christopher

[Myddraal]

A handy site for next time; howf***edismydatabase.com

[AUbicycles]