Fri, 31 Jul 2009

I've heard of code freeze, but please...

The least they could have done is indicate WHICH versions on WHICH hardware. The explanation below reads like the extended version of Excuse Of The Day.

*cough*

From: Customer Services 
Date: xxxxxxxxxxxxxx
To: xxxxxxxxxxxxxxx
Subject: xxxxxxxxx

Re: xxxxxxxxxxxxx Datacentre affected by heavy storm

Dear Valued Customer,

Further to the storm on the XXth of XXXX, we investigated on what  
happened. The Inergen release in XXX has been investigated by  
internal and external experts, and no failures or unexpected  
parameters or tracks have been found. The investigation has covered  
areas such as pressure, temperature, dust, gas, air speed,  
vibrations and turbulence.

Inergen is the industry preferred solution to extinguishing fire in  
data center. However, failed hard drives in connection with the use  
of Inergen in different rooms have been investigated by the  
manufacturer, and their conclusion are:

A sudden temperature change of up to 2-3 degrees within 1 second,  
which some servers have experienced (normal recommendations are max  
5 degrees within 1 hour) have caused some raid controllers, SCSI  
disks and HD to be unstable.

The combination of this instability and a low firmware/driver  
version has caused some of these controllers/disks to fail after a  
period of time (not all failures are recorded at the same time).

The investigation also shows that not all affected disks had  
failures, but the failure in the SCSI/raid has caused the disks to  
fail.

Therefore the conclusion to the failures is that low firmware/driver  
versions are not sufficiently resilient for any Inergen generated  
shift of temperature and an upgrade of firmware/drivers can be  
needed in order to eliminate the chance of failure in the future.

Thus xxxxxxxxxx recommends that all customers ensure that the newest  
supported/tested available version of firmware and drivers are  
installed.

Furthermore, based on the incident, xxxxxxxxxx will continue to  
investigate if change in infrastructure or parameters can reduce the  
impact on the installed hardware in case of any Inergen release.

All related fire extinguishing systems will be back in normal mode  
at end this week.

An incident report has been finalized.

If you have any questions regarding this communication, please  
contact xxxxxxxxxxxx or send an e-mail to customer.services@xxxxxxxxxx.com 

. Please reference above ticket number when you call.

Respectfully,

xxxxxxxxxx European Customer Service Center

Comments? Hah!