First off, if you’re one of my clients, rest assured that this story isn’t about you – even if I’ve graciously terminated our relationship at some point in the past. This is an actual true story from a few years ago.
When I’m working with a new client, we check backups together first. We all need to be on the same page about how often the backups are performed, whether they’re succeeding or failing, and where they’re being written.
It’s not unusual for us to discover problems together. Backup jobs fail, custom scripts miss newly added databases, or mission-critical databases are in simple recovery mode. These things don’t freak me out – we’re all human, and we make mistakes, and after all, I’m not being called in because things are great.
But at one brand new client, things went differently. We discovered that their main production server – housing live financial transactions for their customers – was only backed up once per day, and the backups were written to the same drives that housed the SQL Server databases.
If they lost Windows, the RAID array, the RAID controller, or any number of things, those backups would be useless.
How That Conversation Went
Me: “Do you understand the risks here?”
IT Team: “Yes.”
Me: “Have you explained this risk to management?”
IT Team: “Yeah, but they don’t really get it, and it’s not a priority for them right now.”
Me: “OK, let’s go explain it to them together right now. I have a fun way of explaining things like this.”
IT Team: “Actually, we’d rather not. Let’s focus on the performance issues today.”
Consultants, Here’s Your Sign
The biggest risk for me as a consultant isn’t someone skipping on their bill.
The biggest risk is getting my name associated with a really, really bad idea. I only have one reputation, and I don’t want to be known as “the guy who stood by while Company X lost their entire database.”
Whenever you’re working on a project, don’t think of it as private – think of all of your peers and future customers watching over your shoulder. Can you defend what you’re about to do? If you were called to testify in court about the project, would you feel good about it?
Breaking Up is Easy to Do
Me: “I want to make sure I understand so we’re all on the same page. For you, getting the performance fixed is more important than having backups.”
IT Team: (after much hemming and hawing) “Yes.”
Me: “And I’m going to guess you wouldn’t let me put that in writing and get management to sign off on it. I know that sounds crazy, but I’m an outsider, and here’s the scenario: I could easily sit down at that production server, and while I’m working, someone else could drop a database. It’d be impossible for me to prove I wasn’t the one who dropped it, and it would cost me a fortune in court just to try to defend myself. So I’m even willing to work on this – but only as long as we whip something up in writing saying that the company executives understand the risk, and they indemnify me from all risk.”
IT Team: “Uh, no, we can’t do that.”
Me: “So if you were me, would you take on this risk?”
IT Team: “Wow. When you put it that way, I guess not.”
We shook hands, I walked out, and didn’t bill them a dollar. I didn’t want to have my name associated with that project in any way, shape, or form. A few months later, the company went bankrupt, and I wasn’t the least bit surprised.
You’ve only got one reputation. When a client asks you to do work that you’d be ashamed to admit in public, it’s time to get everybody on the same page, make sure the client won’t change their mind despite your best sales pitch, and then gracefully step aside. They’ll find another consultant willing to do it – there’s always a few – and you’ll find another client.
7 Comments. Leave new
Years ago, I was asked to fix an Access database that I didn’t create.
From the conversation with my client, it was obvious I wasn’t his first choice, or even his second. I was getting the impression that the project had been passed over by several of my colleagues.
Still, I took the job because I needed the work. And, as you can guess, the grumping from the client started within the day. I wasn’t moving fast enough (the same day I took the project), and could I add this, and could I fix that (I had been hired simply to get the database running again), and if I didn’t do all of this, he’d tell everyone in our industry, and I’d never get any clients ever again.
I finally decided this client needed to be fired. I called him up, voiced my concerns, said I was zeroing out his balance with my accounts receivable, and him luck. As I hung up the phone, he was still screaming at me that he’d get even with me. All of this happened within 3 days of me taking the project.
I never heard from him again, and as far as I could tell, it had no negative repercussions with my industry.
There are bad clients out there. I learned I shouldn’t just take any project waved in my face, because I was taking on a world of stress for not nearly enough money.
I learned my time and peace of mind are valuable too, and that I’m not responsible for the client’s mistakes.
Your blog just reminded me of all that. Good lesson!
Ouch. I can believe it. And sadly, somebody’s going to take that work because they’re desperate.
Excellent post and one that I can really identify with.
I’ve just today walked from my current client as I’ve had enough, as I point out critical risks in their data infrastructure and have my solutions ignored or binned.
What moves me to comment is that my client is also in the financial sphere and has this exact backup situation. And although I didn’t react with the same aplomb as you, the BI guys (who own and ‘manage’ the DB) had the same ‘meh’ reaction and quickly pooh-poohed the idea of getting it fixed for various mumbled, unclear reasons.
Hilariously, the single production server (no redundancy or DR/BC capability), which has less memory that my smartphone (no joke), crashes nightly. No amount of pleading and begging has persuaded my client that working to build a scalable fit-for-purpose platform is worthwhile.
It’s my view that some people can’t or won’t be educated, can’t or won’t read best practice whitepapers, can’t or won’t see what’s right in front of them or accept any idea that isn’t their own. And the old adage rings true – you can lead a horse to water but you can’t make it drink.
Great post. Sometimes these scenarios are used as a “test” of consultant competency. I was once brought in for a few days to learn a system and scope out a migration for a longer term project. I noticed patently obvious flaws similar to those you mentioned and asked politely that those shortcomings be part of the scope otherwise I wasn’t willing to bid on the project for the same fears you mentioned.
I found out later that I was the only consultant that made a demand like that, or even pointed out those existing deficiencies. All of the other bids viewed that as being out-of-scope. Even though my bid came in with a much higher “cost” than anyone else’s, I still won the business. I learned later that those deficiencies were a “test” of consultant competency and I was the only one that passed.
Dave – ah, interesting. That sounds like a pretty savvy client.
Been there, bought the t shirt and used it top dry my eyes when a client wanted me to archive a 300GB SQL2K Production database with no backups for thirty days.
My aim when I leave a client is to have the DBA manager as paranoid as me about their data.
SC.
Let’s see now…
(A) Access Databases that don’t work, need fixing, client is a bit of an arse about it. Check Been there, seen that.
(B) Client buying old kit, you specifically get it in writing as such, making clear, it is old, not a good idea and won’t work out well. Kit dies in no time, uh, because it is old. Who do you think was to blame for that?
(C) SQL Databases the world relies on, no backups, simple mode etc. Oh yes, today. Fortunately the reason this was discovered was because the man in charge (and thus has authority to get things sorted) became aware of said issue and signed off fixing it immediately.
(D) No backup, backups. Oh hell yes. I’ve lost track of how many times. Insert also: Dropbox, RAID-0, “I do it when I get time” and so on.
I’ve also “given notice” to my clients in the past. One actually had an argument that we couldn’t do it. We did. He hasn’t been heard from since, we didn’t worry. I do pity whoever has to deal with him now.
Great blog and sadly spot on in too many cases.
Great post, thanks for taking the time to pen it (should we just say “type it” these days?)