Boss Bitch

Between my madman-like posts here I have been trying to lock down a new job.

I got a few good interviews. A decent couple of offers. One really neat damn job that was unfortunately a) very poorly defined b) not too terribly well compensated was at an internationally known facility that anticipates being part of the the most massive non-classified aggregation of data EVER. Instead I am probably going to take a position that is a) pretty doggone generous b) amazingly routine with a medium-large life insurance that almost certainly will outlast almost all of us...

They just put me in charge of approving and scheduling every change to our systems. I get a raise, but damn, this is too much like real responsibility. Not authority, as I don't get to decide what we do, I just get to decide that the req that allows our email to be routed in-the-clear through North Korea may not meet our guidelines and should be sent back for further technical review. Oh, and chances are, I get to be the one implementing the changes, too, as everyone else is busy with their own projects. Sucks being the n00b - I have yet to carve out a fiefdom that insulates me from actually having to do work.

Designated blamee? Do you have a target on your back? Is the company providing red Star Fleet security shirts?

I wish I could say more about how the department is organized, and how I do what I do, but I can't. Let's just say a regimen of icy competence* keeps our department above the blame-game for the most part.

This is, without a doubt, the most interesting and challenging job I've ever had. I love walking the high wire without a net, I love being asked to do two impossible things in one day. I get that kind of stuff here, so I like to brag sometimes. Maybe when and if I move on to the next gig, I can start telling the tale with more detail... =this= is how you run an IT department.

(*We have a guy who's been with the dept. since it was first founded. He can trace an application's packet from the West Coast to the East Coast through the most complex and secure network I've ever seen or even heard of... in his head.)

I am happy for you. I hope it continues to go your way. For me it was quite odd. There were some things that did work reasonably well, but other stuff was appalling. In my last few weeks the report on consolidation sort of pushed me over the edge. To think that my location would be at risk of consolidation because another in the middle of the country hugely overbuilt their datacenter, or an organization deep in the rust belt would be allowed to quarterback such a consolidation just because their own prospects for local utilization were so bleak, or the least competent people in the system would get the nod because they were physically closer to the seat of power was too much to take. Granted, I know none of these things would happen overnight, but laid on top of the disruption that it was clear we would be facing caused me extreme anguish.

If you love your job, howsabout a avatar change? :)

My boss called me yesterday to ask how I was doing. My current boss is a nice guy.

Oh, I still hate everything else.

Quote

I love being asked to do two impossible things in one day.

I thought that the criterion was six impossible things before breakfast ;-)

In the retail world, the holiday shopping season is very important. This should come as a surprise to exactly zero people. Because of the critical nature of the holiday shopping season, a concept known as a "Holiday Freeze" was invented. This is a period of time during which changes to the environment are limited to critical fixes. The intention is to focus on stability. It's a good idea. It would be a great idea if everyone didn't ignore it. One team in particular made a change last week... at the end of the day on Wednesday, conveniently, which is when I left for my 4 day weekend. When I got back today I learned that "shit done be broke." At first it didn't look like it would involve me at all. This was good because I had a LOT of work to catch up on after my long weekend. My joy was short-lived as another problem creeped up that seemed related and did involve me... at least tangentially. In troubleshooting, I discovered some irregularities that raised quite a few questions. That's when I learned about the change on Wednesday and realized that it was, in fact, the cause of both problems. As if by coincidence, it was at that moment that I was asked to join the "Command Bridge" that had been set up to deal with the first issue. I walked into the room, asked them when, exactly, the change had been made. I then asked them why they didn't just roll back the change. You would think I had just given a cave full of Neanderthals their first demonstration of fire. Seriously. This was a room full of people (several of which are higher on the ladder than I am) and after many hours of brainstorming, they hadn't come up with that one on their own?

I almost wish I had kept my mouth shut because what followed was another few hours of bureaucratic tap-dancing as the culprits desperately tried to find any way to deflect blame. I left for a while and was about to leave the office to go get some dinner (I had skipped lunch because of this) when they called my boss (who was leaving at the same time and was walking down the hall with me) saying that they needed us both back in the Command Bridge. They had successfully dredged up a red herring and had a senior exec convinced that there was some degree of shared blame. Again, I had to inject the obvious. The 'change' they say that I had made that contributed to the problem was made back in February (and had been fully tested). It's true that my change did, inadvertently, increase the load on our DNS servers, but everything functioned fine for 8 months. In a somewhat snarky tone (hey, I was hungry) I said, "my change was made 8 months ago. This problem appeared on Wednesday. So how does that point to my change as the cause of this problem?" The senior exec looked at me with a sort of "that's-a-damned-good-point" look... and followed it up with something along the lines of, "that's exactly what I'm thinking. That may be a problem that we need to address but first we need to address the change that happened on Wednesday that triggered these immediate failures."

This was my first chance to contribute to a discussion with this exec. He's a sort of no-bullshit guy (his first all-team meeting after we were brought under him in the org chart included a brief tidbit about the correct way to drink tequila) so I'm hoping that my demeanor in the Command Bridge had the appropriate impact.

If I was in any company other than a navy (or starfleet) that had a "Command Bridge," I would quit. I admire your stance, CS, and I hope that the executive made note of it.

One for the Command Bridge? Hmm.

A friend of mine from high school is the lead web developer and his Facebook status currently implies problems with a planned deployment. It's completely separate from the retail stuff... in fact, even though we both work on campus, I haven't seen him since high school.

Oh, by the way... a "Command Bridge" is a temporary thing that they do when major problems come up. They take over a conference room and set up a continuous conference call for the duration of the "emergency."

Ahh... this explains it.

I forget that not all of IT types are as up to their armpits in "Network Operation Center" type "conference calls" as I tend to be. If I had a nickel for every CHANGE that was plunked in during a freeze I would no long have to work -- if I got $10 for everytime that one of those "freeze exception" changes did cause some problem I could probably also live like a minor deity in some countries...


It was worst on the consumer side of a certain retailer that I worked for, but the financial services places I worked have been quite reckless too. Tough thing is that in financial world you can just about be certain that some dunderhead will fret that rolling back a failed change BY ITSELF will constitute what the IT audit types steeped in SarbOx term a "significant operational risk" even when there is no true exposure (due to the magic of doubly-redundant systems...) -- I have actually shouted "if we never 'failback' THAT is what constituents a TRUE operational risk in the same way that a belt & suspenders kind of guy really only risks exposing his boxers (or worse) once EITHER the belt OR suspenders FAIL.



Edited 1 time(s). Last edit at 10/13/2008 10:44PM by stan adams.

Very recognisable CS, I have just sent off the Acceptance environment of our systems to India (trip was cancelled 8 hours before we were going to leave) only to get a reply requesting for the database creation scripts. I replied that they surely must have them as they are the frikkin' developers, they bluntly told me that the Acceptance (or Production) environments were further ahead of their own test environments.

>I could probably also live like a minor deity in some countries...

You'll always be a minor deity to me, Stan.

I've been up for 26 hours.

I'm going home now.

But you can rest easy knowing that your productivity has gone straight to strengthening the economy.

http://finance.google.com/finance?catid=59492966

Oh, wait, never mind. Just go to bed, and maybe dream about how you've helped something :)

Okay, time for some overdue gripes.

Last week we had a big meeting with my boss's boss's boss. He brought everybody together who works under him and told us that we "rock" ... "but" ... yeah. You can see where this is going. He said that we're "not quite world-class" but that we're "right on the edge." Then he spent two hours telling us all the things that we need to do in order to become "world-class".

At the end of the meeting my team was furious. Everything he talked about is exactly what our team does. He gave examples of failures but they were all from other teams and they were mistakes that our team does not make. It's the reason we have been able to function with such a limited staff. What this told us is that:

A) He has no fucking clue what our team does.
B) We're apparently the best team he has but he doesn't care.

To make matters worse, everyone on our team got their ratings this year and once again, we were all ranked "consistent with peer group" which is the lowest rating you can get before you go into the "bad" ratings. So if we're already doing exactly what he expects the rest of his teams to do, why didn't our team get ranked higher?

Whatever.

Then comes this patching emergency. I spent the entire night at my desk doing something that wasn't possible for the rest of his teams. I was prepared. I knew that this sort of thing was going to happen and had tested, documented processes ready to handle it. I managed to patch our entire retail chain with ZERO impact on operations. The corporate team won't be done patching their devices until Monday... oh, and while they were patching the 9,000 devices they did get around to last night, they managed to COMPLETELY FUCK UP. They aren't supposed to touch my deployment servers. My team patches those ourselves. Of course they don't have tested, documented processes for this kind of situation and their lack of organization caused them to reboot my deployment servers IN THE MIDDLE OF MY DEPLOYMENT! Luckily, my processes are rather robust and I was able to recover (it set us back about 20 minutes). They also managed to whack the reporting system that corporate uses and that set them back an hour and a half as they tried to recover the database that was corrupted by the unexpected reboot.

Fast-forward to today. As people are getting into the office in the morning and learning about what we did, I couldn't help but wonder what that jerk was thinking about my team now. Might he be considering us "world-class" yet? He'd damned well better think so because HIS boss sent out an email this morning recognizing our, "world class capability/performance."

Boo-ya. We gots da mad skillz, y0.