Server Error - Forum Restored from Backup - Posts Lost

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Mon Nov 08, 2010 9:13 pm

Hello,
I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.

This means that posts have been lost (weekend) and I apologise for this lost and hope that you can forgive this - both the host and myself have worked to find the quickest most acceptable solution to getting this back online. It also means that brand new members who have registered will need to register again.

Comments are welcome... and I am currently looking to get the main site online as well.

Cheers
Christopher
Cycling is in my BNA

User avatar
The Womble
Posts: 3395
Joined: Thu Apr 01, 2010 6:46 pm
Location: Brisbane QLD
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby The Womble » Mon Nov 08, 2010 9:16 pm

Ooh yeah! im the first poster yeah!

You can still access most threads from the Weekend Chris

User avatar
peter
Posts: 1411
Joined: Sun Nov 25, 2007 7:39 pm
Location: sydney

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby peter » Mon Nov 08, 2010 9:35 pm

I thought it's Groundhog Day!

By the way, just got SQL error again.

Hope the host can track down the cause and provide a fix.
Last edited by peter on Mon Nov 08, 2010 9:38 pm, edited 2 times in total.

User avatar
twizzle
Posts: 6402
Joined: Wed Aug 13, 2008 11:45 am
Location: Highlands of Wales.

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby twizzle » Mon Nov 08, 2010 9:37 pm

peter wrote:I thought it's Groundhog Day!

By the way, just got SQL error again.
+1 to more SQL errors, and I'm missing PM's from the middle of last week. :(

Edit: O.K., it was probably from late Friday that I lost messages. My memory is sh!te.
Last edited by twizzle on Mon Nov 08, 2010 9:38 pm, edited 1 time in total.
I ride, therefore I am. But don't ride into harm's way.
...real cyclists don't have squeaky chains...

User avatar
CommuRider
Posts: 5053
Joined: Sat Sep 25, 2010 6:16 pm
Location: Sydney

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby CommuRider » Mon Nov 08, 2010 9:37 pm

Oh well, at least I managed to read the Gong posts....
Amateur oenologist and green-friendly commuter.

User avatar
schroeds
Posts: 879
Joined: Tue Jul 14, 2009 9:58 pm

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby schroeds » Mon Nov 08, 2010 9:56 pm

(sob) my hairy ride thread has disappeared and it was shaping up nicely :cry:
Not so much a cyclist...more of a sit down comedian

User avatar
queequeg
Posts: 6477
Joined: Thu Dec 17, 2009 10:09 am

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby queequeg » Mon Nov 08, 2010 11:05 pm

AUbicycles wrote:
I have turned the forums back on - out of maintenance mode - there was a MySQL error and it ended up corrupting the database and rendering it useless so I asked the host to restore from a backup.
This is when DBAs can justify their salary :-) I've had my fair share of these in the last 15 years, but unfortunately the data has been a bit more "critical" than forum posts. I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL :-)
'11 Lynskey Cooper CX, '00 Hillbrick Steel Racing (Total Rebuild '10), '16 Cervelo R5, '18 Mason BokekTi

User avatar
Max
Posts: 2895
Joined: Wed Jun 18, 2008 1:29 pm
Location: Brisbane
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby Max » Tue Nov 09, 2010 6:26 am

queequeg wrote:I have a nasty knack of finding new bugs in database software, so I promise to stay away from MySQL :-)
What the! You stay the hell away from our forums! Who knows just how much of a hand your "influence" had on this event!! ;)

Seriously - thankyou Chris for all your efforts in getting the site back up and running. We lost some posts, so be it. Better than losing the entire board.

:)

Max
One of the best things about bicycle commuting is that it can mitigate the displeasure of having to go to work. - BikeSnobNYC
Cycling is sometimes like bobbing for apples in a bucket full of dicks. - SydGuy

User avatar
Comedian
Posts: 9166
Joined: Mon Aug 09, 2010 7:35 pm
Location: Brisbane

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby Comedian » Tue Nov 09, 2010 7:00 am

Having been there too I appreciate your efforts!

User avatar
Apple
Posts: 3513
Joined: Sun Apr 11, 2010 4:00 pm
Location: sydney

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby Apple » Tue Nov 09, 2010 7:01 am

Yes, I had a PM waiting for me to read, it could have been great gossip and who ever sent it please send it again. :D

Good job on getting it fixed, I am sure it wose a son of b%^ch to fix.
A successful man is one who makes more money than his wife can spend. A successful woman is one who can find such a man.
Speak your mind,Those that mind dont matter, Those that matter dont mind!!

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Tue Nov 09, 2010 9:39 am

Thanks everyone.
My contact with websitemanagers was also travelling interstate and so took time on the fly to help support and resolve. Credit.
And... we have been doing very very well, no issues for a while.

Still interested if we can get to the bottom of this and see what preventative methods we can get into place.
Cycling is in my BNA

User avatar
Saro28
Posts: 561
Joined: Wed Jun 02, 2010 5:36 pm

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby Saro28 » Tue Nov 09, 2010 7:04 pm

top job blokes, good luck finding and fixing the prob' (cue Thumbs up emoticon here)

Andy
Fight till the end n never give up. Cadel Evans 2012

User avatar
snark
Posts: 581
Joined: Wed Oct 01, 2008 2:35 am
Location: Geelong, Vic

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby snark » Tue Nov 09, 2010 11:29 pm

AUbicycles wrote:Still interested if we can get to the bottom of this and see what preventative methods we can get into place.
For the preventative part, how big is a dump of the database (compressed or zipped)? Maybe this can be backed up multiple times a day, and if it compresses well you could keep a reasonable number of backups, perhaps off-site.

Probably stuff you've thought about already, but maybe worth a mention...

Cheers,
Simon.
Phil Liggett, TdF 2011, Alpe-d'Huez: "I reckon tonight in hindsight he may have won the Tour de France tomorrow."

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Tue Nov 09, 2010 11:51 pm

The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.
Cycling is in my BNA

dclnmurray
Posts: 143
Joined: Fri Aug 13, 2010 6:13 am

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby dclnmurray » Wed Nov 10, 2010 6:08 am

I do a bit of database admin work so know the hassles involved. If the client doesn't use any database replication (database mirroring etc) our standard is using maintenance plans to do a database backup in the early hours of the morn and transactional hourly backups in between. Assuming the DB is not in simple mode. This allows recovery model to the most recent transaction backup although I am not aware of how MySQL works in this regards.

Anyway, appreciate the work done.

Dave
Image

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Wed Nov 10, 2010 10:16 am

Thanks Dave... good thoughts.
One thing that caught us out were the MySQL errors and not being able to pinpoint exactly when the errors may have crept in.
Cycling is in my BNA

User avatar
queequeg
Posts: 6477
Joined: Thu Dec 17, 2009 10:09 am

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby queequeg » Wed Nov 10, 2010 10:24 pm

AUbicycles wrote:The biggest part of this forum database is 600 MB (485,000 posts) and it tends to be a bit of a load doing the backups. Once we get to the bottom of it I will have a look at the options. Thanks for the feedback.
So it is quite a small database then ;-)

I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!

I don't have any experience with MySQL, but most of the databases I deal with these days are sized in Terabytes. Our backup strategy is a mixture of technologies, depending on the specific requirements. The most critical system all run standby databases at secondary sites with "near real time" syncing of transactions. of course, if you get a logical corruption in the transaction logs then it does not help. In those cases it is restore from backup and roll-forward to the last transaction before the corruption.

Personally, you have not lived until you have been working on a Severity 1 Database Corruption issue with the Database Vendor for 72 hours straight, trying to isolate the corrupt blocks and somehow bypass them so that the database instance stops crashing. Life as a DBA is never boring :-)
'11 Lynskey Cooper CX, '00 Hillbrick Steel Racing (Total Rebuild '10), '16 Cervelo R5, '18 Mason BokekTi

User avatar
twizzle
Posts: 6402
Joined: Wed Aug 13, 2008 11:45 am
Location: Highlands of Wales.

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby twizzle » Fri Nov 12, 2010 7:47 pm

queequeg wrote:I had to do a rebuild of an index on a single database table today. The index alone was 35Gb, and the table had 415 million rows in it!
I did performance reporting on a system that had ~ 1 billion rows in the main table, despite weekly culling of old data. Mainframe VSAM... then they tried running it on DB2 instead. Didn't perform anymore. :roll:
I ride, therefore I am. But don't ride into harm's way.
...real cyclists don't have squeaky chains...

User avatar
CommuRider
Posts: 5053
Joined: Sat Sep 25, 2010 6:16 pm
Location: Sydney

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby CommuRider » Fri Nov 12, 2010 7:51 pm

Had the SQL error around 5pm. Is it stable again or is this a brief respite?
Amateur oenologist and green-friendly commuter.

User avatar
The Womble
Posts: 3395
Joined: Thu Apr 01, 2010 6:46 pm
Location: Brisbane QLD
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby The Womble » Fri Nov 12, 2010 7:54 pm

Those warrrr the days. I 'member havin't eat a handfull 'coal, trudge down t' mineshaft t' office, work with 14 billion lines of data whilst sittin' on t' chair covered in broken glass...

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Sat Nov 13, 2010 10:07 am

Updates are going here. This issue was related to the weekend problem as a (failed) attempt to improve performance. Host was onto it quickly and more errors are not expected.

Cheers
Christopher
Cycling is in my BNA

Myddraal
Posts: 282
Joined: Thu Jul 08, 2010 2:13 pm

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby Myddraal » Mon Nov 15, 2010 9:20 am

A handy site for next time; howf***edismydatabase.com

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15579
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Server Error - Forum Restored from Backup - Posts Lost

Postby AUbicycles » Tue Nov 16, 2010 9:36 am

:)
Cycling is in my BNA

Who is online

Users browsing this forum: No registered users