Thursday, April 3, 2014

Mirror, Mirror

This week, I got to meet with a new client who is having trouble with database maintenance and performance. One of the things the client had mentioned was that database objects were disappearing on a fairly regular basis and nobody quite knew why. This client has two Database Administrators (hereafter referred to as Walter and Gladys) who weren't quite sure what was going on.

Gladys I didn't get to meet. She works from home for health reasons. She's a quiet, passive sort of creature, really quite harmless, but utterly unable to counterbalance Walter. Walter, I was warned, is a thing to behold.

Walter talks too fast, and too strongly. He's difficult to budge off a subject and difficult to engage. He dominates conversations with nothing in particular to say. He doesn't know what he's doing and as a result, the system he's charged with keeping running smoothly careens from crisis to crisis. During one of those crises, his boss appeared in his cube to ask how the remedy for said crisis was progressing and found Walter watching cartoons on his iPad.

I met Walter and felt sorry for him. I like people with crazy energy levels, and I have an easier time masking my energy than Walter does. I tend to get excited when explaining a technology or a solution, and I believe the general impression this gives is that I am "enthusiastic," even "passionate" about my work. And I am. I couldn't do this if I didn't find it intriguing. I wouldn't be able to stick with slogging it out to learn this if I didn't find it genuinely pleasurable to tinker with databases. So Walter's oddness, his rambling, his energy, they didn't seem that strange to me.

When he was taking my history, my therapist asked me if I have any physical twitches or tics. I showed him how I fold my hands into each other, fingers curled with each other like a greek key, to conceal the movement. "You feel you have to hide this?" he asked.

And I thought of all the times a co-worker has grabbed my drumming hand, stopped my bouncing knee, taken away my pen, hit me in the shoulder to warn me not to talk, the evaluations that have mentioned my doodling while I listen and my doodling while I talk, and one-on-ones with a boss who has tried to explain to me that I'm passionate and that's a good thing, but... Before having a name to put to this, it was hard to explain even to myself why it was so necessary to me to behave in ways that were so clearly not helpful to me.

Looking around the conference table, I could see how strange Walter's behaviors appeared to everyone else. I could see that his boss found it difficult to tolerate Walter's presence. I could see that my colleagues found it difficult to be tactful about his "input."

Sometime early that afternoon, Walter gave an instruction to interrupt a process that refreshed database objects on the public-facing production server. Soon after that came the first notice that a table was missing. The first commandment of Database Administration is "first, lose no data." The next 99 commandments are variations on the first. It is the cardinal, absolute, unvarying taboo of database administration to allow data to disappear, and the absence of that table meant that the public website would no longer be functional. Walter got the notice as he was leaving the building, and decided that it did not constitute enough of an emergency to interrupt his afternoon plans.

At 3:00, completely unrelated to the database problems, smoke had been detected on our floor and the fire department requested that we vacate. At that point, we didn't know anything was going wrong with the databases and settled in to the benches in the light rail station to wait for early trains out to the suburbs.

At 5:00, I had just gotten to my hotel. I checked in, grabbed some delivery menus, took a good hot shower and was getting ready to settle in for a night spent with an order of Shrimp with Mixed Veg and Lord Jim when my phone buzzed. Walter's boss had decided to open his checkbook and get the consultants on the phone to help get things back online.

The next seven hours were pretty goddamned awful.

Walter wouldn't follow directions. He wouldn't wait for directions. He said things like "do you think I should press this key? I pressed it." He was confounding, obstructive and utterly dominant. Nobody else could get a word in. My colleague was listening intently, as was Walter's boss, trying to intervene to offer what correction they could. It was fortunate that they were, because after the first time I offered the instructions on how to solve the problem and having supplied a script to do the job, I was not listening to the call with more than half an ear. I spent those hours "multitasking" - working intently enough on scripting other work for the client that I was almost entirely unaware of the content of the conversation. I was fortunate that, in this case, this was seen as a good thing. I could just as easily have been called to task and asked to pay close attention without actively participating, and that would have been almost impossibly difficult for me.

At 1:17, it looked like things were back up and running again. We agreed to reconvene at 4:30am to check on the site and make sure it was working before the rush hour traffic hit. At 4:26, Walter sent an email saying that everything looked good and he was going back to bed. At 4:31, my colleague and I had confirmed that the database was still not functional and had last gotten data input shortly after 1:00 the previous afternoon. We reassembled without Walter and had restored the database properly in about 15 minutes.

I got to hear the next day my colleagues venting freely about Walter. I got to hear them speculating about whether he would still have a job by the end of the day. I understood that - Walter had committed a series of cardinal sins and wasted a great deal of time and money in the process. "I had to tell him over and over again," my colleague said "to wait and not do the latest fool thing he'd decided to do. We'd sent him a script and he screwed it up because he wouldn't follow it. And then," and here he gave a carnivorous grin, "when it finally looked like it was working, he wanted to fiddle with it, and Jennifer just about came through the phone at him, telling him that that's exactly how to screw it up." I blushed and realized that I had been impulsive, fast-talking, dominating. Fortunately, I had spent most of the previous hours with my line on mute; also, fortunately, my statement and manner of delivering it was in that case accepted as justifiable.

I don't mean to suggest that I am as dysfunctional as Walter. I have a keener sense of triangulating to my strengths than he does (for one thing, I wouldn't ever take a job whose primary function is maintenance, as he has done, because I know that my temperament isn't suited to it). But I sat there the next day, listening to the office outrage about his incompetency, smiling, saying little, with my fingers curled together, like a greek key.


  1. Well, how were the tables getting dropped?

    1. Our best half-assed guess is that one of the DBAs was making changes directly to the subscriber database, which, when the subscriber detected that it was out of sync in ways that weren't expected, pushed the subscriber into doing a full object synchronization (refresh). Then they saw that that synchronization was causing blockage and decided to kill it by restarting the publisher's server. But the reason for the blockage was that replication was completely overhauling the tables, so when they killed it without giving it any way to roll back... the tables were just gone. When we did the postmortem, we counted nearly 100 objects total missing from databases on each public-facing server, more than half of them tables.