What the H happened! (site down, major hardware failure)

Tell us what you think! This forum is here to serve the Cougar hobby, we need your help to make it better.
Post Reply
User avatar
xr7g428
Site Admin
Site Admin
Posts: 4305
Joined: 23 Dec 2010, 21:02
Location: Cave Creek, AZ
Contact:

What the H happened! (site down, major hardware failure)

#1

Post by xr7g428 » 04 Jan 2014, 21:14

If you had been trying to get on this site and were having problems, I can assure you it is entirely not a problem with your computer. It was all on the server side.

In spite of having three layers of back up and redundancy, some times it really does hit the fan and last night starting at about 10PM PST things started to get down right nasty. At 3 AM the site was completely dead. Not only were the sites dead, but also the email system, the Private Message system and more.

What happened was that we suffered a catastrophic hardware failure. The entire contents of the server we are on had to be reinstalled from back ups. And that is the good news.

The bad news is that the server had been feeding corrupted files to the multiple layers of back up, and the newest clean back up was from December 29th. At first it looked like we were going to lose everything, that had been posted from that point forward. Fortunately, the host we rent the server space from was able to provide Jay with selected files including our own back up file from 3AM on the morning of the 3rd. So we did lose a few things, but a lot less than originally thought. The thread that I think was hurt the worst was the Are You Coming To Barrett Jackson thread. If you posted there, you might want to replace your post.

Due to some very very good work on Jay's part we are back and almost completely intact. Jay suggested that I need to stock up on Chapstick as I have some serious butt kissing to do at the Hosts location as they really did come through for us in a big way. I am not much of a kisser so this should not be much fun for them either.

Anyway, I am sorry about the interruption of service and the general confusion as we fired the board up, took it down and fired it up again, and I know several of you posts disappeared in the process. I'd like to say that it will never happen again (this is only the 2nd time in almost twenty years we have seen this failure) but sometimes the bear does eat you.

Thanks for everyone's patience and understanding!
Bill Basore, Editor / Publisher
Legendary Cougar Magazine
Currently in the Cat House
'67 XR7 GT 390 4 speed, AC, AM FM, Lime Frost Green
'68 XR7-G 428CJ C6, Tilt-Away, AM, Black Cherry
'68 XR7-G 390 4 speed, Sunroof, Cardinal Red
'68 XR7 GT-E 427 C6 AM Cardinal Red
'68 XR7 resto mod 351W, soon to be AOD, Black Cherry

User avatar
CatVert68
Registered Member
Posts: 1239
Joined: 18 Jan 2011, 20:09
Location: Central MA
Contact:

Re: What the H happened! (site down, major hardware failure

#2

Post by CatVert68 » 04 Jan 2014, 22:37

Bill, I've pulled a few all-nighters getting IT systems back on-line so I know from experience that some failures don't care how many layers of protection you have. We appreciate the speed with which you and Jay go the site back up and running. It was quick enough that I didn't even realize there had been an outage! Great work to all those involved in the recovery effort.
======
CCOA Member 8302

1968 Cougar XR-7 GT, Madras Blue & Parchment w/ Parchment vinyl top
1969 Corvette L-48 4-speed LeMans Blue over Bright Blue

User avatar
xr7g428
Site Admin
Site Admin
Posts: 4305
Joined: 23 Dec 2010, 21:02
Location: Cave Creek, AZ
Contact:

Re: What the H happened! (site down, major hardware failure

#3

Post by xr7g428 » 05 Jan 2014, 10:27

Bill, I thought you and Leon would both be smiling knowing what we were up against. Probably more IT folks on here than I know about, to. Anyway, we seem to be running smoothly now. Jay really deserves all the credit.
Bill Basore, Editor / Publisher
Legendary Cougar Magazine
Currently in the Cat House
'67 XR7 GT 390 4 speed, AC, AM FM, Lime Frost Green
'68 XR7-G 428CJ C6, Tilt-Away, AM, Black Cherry
'68 XR7-G 390 4 speed, Sunroof, Cardinal Red
'68 XR7 GT-E 427 C6 AM Cardinal Red
'68 XR7 resto mod 351W, soon to be AOD, Black Cherry

User avatar
69XR7Ragtop
Registered Member
Posts: 390
Joined: 25 Dec 2010, 02:26
Location: Around FT. Bragg, NC

Re: What the H happened! (site down, major hardware failure

#4

Post by 69XR7Ragtop » 05 Jan 2014, 13:30

Bravo gentlemen, very good recovery time and work! :beerchug:
Thanks for all you do for us.
_______________
Daryl

Previous: 69 XR7 Hardtop, 351W, FMX
Current: 69 XR7 Convertible, Tweaked 351W, TL 4 Speed (Almost completed restoration--drive all the time, but you are never really done!)
Current: 70 XR7 Convertible, 351C, TL 4 Speed Grabber blue (Yet to be restored--waiting patiently)

User avatar
leonbray
Supporting Member
Supporting Member
Posts: 955
Joined: 19 Jan 2011, 00:01
Location: Brisbane, Australia

Re: What the H happened! (site down, major hardware failure

#5

Post by leonbray » 05 Jan 2014, 21:19

Bill.

I didn't respond because I remember too well the lost holiday weekends and long nights from failures of software vendors and service providers and secure 24x7 environments that can never go down.

While I might have to go back to the grind soon, I like the fact that my current challenge is how to replace the exhaust gasket on the 428 (and yes I know that 428's shouldn't have exhaust gaskets)

Failures like the one you just had just reinforce my view that service level agreements aren't worth the electrons they agitated in theirm preparation and failures of 428 exhaust gaskets just reinforce my desire to go back to working on VWs and Porsches

Thanks for overcoming
Designed to be driven

User avatar
xr7g428
Site Admin
Site Admin
Posts: 4305
Joined: 23 Dec 2010, 21:02
Location: Cave Creek, AZ
Contact:

Re: What the H happened! (site down, major hardware failure

#6

Post by xr7g428 » 05 Jan 2014, 22:48

Leon: Amen brother!
Bill Basore, Editor / Publisher
Legendary Cougar Magazine
Currently in the Cat House
'67 XR7 GT 390 4 speed, AC, AM FM, Lime Frost Green
'68 XR7-G 428CJ C6, Tilt-Away, AM, Black Cherry
'68 XR7-G 390 4 speed, Sunroof, Cardinal Red
'68 XR7 GT-E 427 C6 AM Cardinal Red
'68 XR7 resto mod 351W, soon to be AOD, Black Cherry

User avatar
CatVert68
Registered Member
Posts: 1239
Joined: 18 Jan 2011, 20:09
Location: Central MA
Contact:

Re: What the H happened! (site down, major hardware failure

#7

Post by CatVert68 » 06 Jan 2014, 10:17

leonbray wrote:
Failures like the one you just had just reinforce my view that service level agreements aren't worth the electrons they agitated in theirm preparation and failures of 428 exhaust gaskets just reinforce my desire to go back to working on VWs and Porsches

Thanks for overcoming
In over 30 years in the IT industry, I don't think I've ever encountered a vendor who was willing to be held financially accountable for their SLA's. To me, they are nothing more than a marketing document until there is some vendor skin attached to meeting them.

I've been out of the IT organization for 7 years now. It's a lot more fun to train bright, energetic 20-somethings on how to sell to IT than it ever was to do IT. And it probably beats dealing with 428 gaskets as well, if for no other reason than that half of my trainees are recent co-eds entering Sales. It's like an eye candy factory around here!

But I do feel for you, Bill and Jay. And Leon, I'd also prefer working on VW's and Porsches to going back to 24-hour pages and system recovery events.
======
CCOA Member 8302

1968 Cougar XR-7 GT, Madras Blue & Parchment w/ Parchment vinyl top
1969 Corvette L-48 4-speed LeMans Blue over Bright Blue

User avatar
leonbray
Supporting Member
Supporting Member
Posts: 955
Joined: 19 Jan 2011, 00:01
Location: Brisbane, Australia

Re: What the H happened! (site down, major hardware failure

#8

Post by leonbray » 06 Jan 2014, 15:10

CatVert68 wrote:
leonbray wrote:
Failures like the one you just had just reinforce my view that service level agreements aren't worth the electrons they agitated in theirm preparation and failures of 428 exhaust gaskets just reinforce my desire to go back to working on VWs and Porsches

Thanks for overcoming
In over 30 years in the IT industry, I don't think I've ever encountered a vendor who was willing to be held financially accountable for their SLA's. To me, they are nothing more than a marketing document until there is some vendor skin attached to meeting them.
I worked on a couple of accounts with EDS where we would provide customers with compensation for SLA's we didn't meet. The Texas based integrity of EDS was far more satisfying than the west coast ethics of the current owners.
Designed to be driven

User avatar
CatVert68
Registered Member
Posts: 1239
Joined: 18 Jan 2011, 20:09
Location: Central MA
Contact:

Re: What the H happened! (site down, major hardware failure

#9

Post by CatVert68 » 06 Jan 2014, 15:42

leonbray wrote:
I worked on a couple of accounts with EDS where we would provide customers with compensation for SLA's we didn't meet. The Texas based integrity of EDS was far more satisfying than the west coast ethics of the current owners.
Leon, did their financial accountability extend beyond crediting back the prorated cost of the SLA for the period of the outage? That's what most vendors offered when pressed and it typically amounted to peanuts compared to the actual cost of the outage.

I only ever dealt with EDS on projects and SLA's in APJ and EMEA, since we didn't have IT resources in those regions at the time. But I can see where Ross Perot would have operated a bit differently than most.
======
CCOA Member 8302

1968 Cougar XR-7 GT, Madras Blue & Parchment w/ Parchment vinyl top
1969 Corvette L-48 4-speed LeMans Blue over Bright Blue

User avatar
leonbray
Supporting Member
Supporting Member
Posts: 955
Joined: 19 Jan 2011, 00:01
Location: Brisbane, Australia

Re: What the H happened! (site down, major hardware failure

#10

Post by leonbray » 06 Jan 2014, 17:10

CatVert68 wrote:
leonbray wrote:
I worked on a couple of accounts with EDS where we would provide customers with compensation for SLA's we didn't meet. The Texas based integrity of EDS was far more satisfying than the west coast ethics of the current owners.
Leon, did their financial accountability extend beyond crediting back the prorated cost of the SLA for the period of the outage? That's what most vendors offered when pressed and it typically amounted to peanuts compared to the actual cost of the outage.

I only ever dealt with EDS on projects and SLA's in APJ and EMEA, since we didn't have IT resources in those regions at the time. But I can see where Ross Perot would have operated a bit differently than most.
The penalties in some cases related to business impact, but as a minimum was equivalent to the cost of providing the service during the outage
Designed to be driven

Post Reply