Singapore!

This post is seeing some activity after a very long time now.
As much as I think of posting sometimes, it never materializes.

Maybe thats why this article got through as it was originally designed as an email to my close friends.
For the background Information,
I am currently working in TCS posted in Singapore.
My Client is Singapore Airlines, the airlines which has been wining the bet airline in the whole world for more than 6 years in a row! (followed by emirates)


Well, here is what I wrote today ........................




I am in the middle of history creation here!
but unfortunately this will be an infamous incident.

All S'pore Airlines Systems are going down one by one.
It started with a few systems and some connection problems and is slowly becoming unstoppable...

Being at the client side I am able to realize the full potential and the gravity of the situation, which I am sure I would not have felt sitting in TCS office offshore.

The place I sit in is called the Computer Center Building(CCB), which is the backbone of all computer applications of SIA. In short if it is in computers and related to SIA, CCB is responsible.
And this entire building is in chaos since afternoon!

TCS alone maintains some 300 applications of SIA 80% of which are down. Add to this fact that here I sit with people from 3 other different IT companies like emphasis etc. all reeling under the issue.

Many of these are critical systems, which have severity 1 systems, and currently all applications are on max alert and severity 1 levels.
DBA's and Coverage people are afraid to breathe I think.
Lucky my application can have only sev. 3 tickets max!

Applications like SIA's online ticket booking through website is down since morning.
Just to give you a simple idea, every hour this receives more than 100 ticket bookings through users booking online.
Its already bout 6 hours down and calculating the loss due to this comes out to be huge.

But then, this is just one application, things like cargo's, refunds, a vivid array of critical and revenue generating app's are down. Even trying to calculate the loss which S. Airlines will face due to this massive outage fails my maths.

Sometimes we hear in the news, some company faced millions in losses due to some crash or virus attack...
Well this is it...and I am sitting in the calm eye in the middle!

Talking bout news, you will people will probably hear all this in the news tomorrow ie if nothing can be done tonight.

Actually IBM provides us the support for the technical backbone, and last I heard, no one was sitting in the entire IBM floor above us.

Another Goliath in the making is data corruption.
On average here every application is connected to 4 more applications and if production data for one goes into corruption, the relational matrix tells me the whole SIA world will be thrown into a tizzy.
Secondary people have started the maths and logic for things to be done in case of corruption which is very likely in some applications in case of a network failure such as this.

Everywhere people are huddled and discussing the implications of this.
It may seem some people are having fun and enjoying...No system no support right?
People are advised to go home! Why? So that they can come back as soon as there is any good news to work out the repercussions.

Systems should be coming up one by one starting today midnight(The date change is important I gathered.) and people will be called in one by one from their beds. (By the way I don't have a coverage mobile too!!!)

For next 2-3 days people will be on 24/7 support. People on coverage and sev 1 tickets are saying things like no sleep for the next 48 hours or so.

I am not much involved as I am only responsible for the UAT server and that is still up !
(I take a lot of pride in saying this although it is not up because of me! Its simply on a different server.)

But still, I am witnessing something great which hasn't before (not of this scale anyway) and hopefully wont happen again.
(If they don't let go of me that is!!!)