First off, if you’re one of my clients, rest assured that this story isn’t about you – even if I’ve graciously terminated our relationship at some point in the past. This is an actual true story from a few years ago.
When I’m working with a new client, we check backups together first. We all need to be on the same page about how often the backups are performed, whether they’re succeeding or failing, and where they’re being written.
It’s not unusual for us to discover problems together. Backup jobs fail, custom scripts miss newly added databases, or mission-critical databases are in simple recovery mode. These things don’t freak me out – we’re all human, and we make mistakes, and after all, I’m not being called in because things are great.
But at one brand new client, things went differently. We discovered that their main production server – housing live financial transactions for their customers – was only backed up once per day, and the backups were written to the same drives that housed the SQL Server databases.
If they lost Windows, the RAID array, the RAID controller, or any number of things, those backups would be useless.
How That Conversation Went
Me: “Do you understand the risks here?”
IT Team: “Yes.”
Me: “Have you explained this risk to management?”
IT Team: “Yeah, but they don’t really get it, and it’s not a priority for them right now.”
Me: “OK, let’s go explain it to them together right now. I have a fun way of explaining things like this.”
IT Team: “Actually, we’d rather not. Let’s focus on the performance issues today.”
Consultants, Here’s Your Sign
The biggest risk for me as a consultant isn’t someone skipping on their bill.
The biggest risk is getting my name associated with a really, really bad idea. I only have one reputation, and I don’t want to be known as “the guy who stood by while Company X lost their entire database.”
Whenever you’re working on a project, don’t think of it as private – think of all of your peers and future customers watching over your shoulder. Can you defend what you’re about to do? If you were called to testify in court about the project, would you feel good about it?
Breaking Up is Easy to Do
Me: “I want to make sure I understand so we’re all on the same page. For you, getting the performance fixed is more important than having backups.”
IT Team: (after much hemming and hawing) “Yes.”
Me: “And I’m going to guess you wouldn’t let me put that in writing and get management to sign off on it. I know that sounds crazy, but I’m an outsider, and here’s the scenario: I could easily sit down at that production server, and while I’m working, someone else could drop a database. It’d be impossible for me to prove I wasn’t the one who dropped it, and it would cost me a fortune in court just to try to defend myself. So I’m even willing to work on this – but only as long as we whip something up in writing saying that the company executives understand the risk, and they indemnify me from all risk.”
IT Team: “Uh, no, we can’t do that.”
Me: “So if you were me, would you take on this risk?”
IT Team: “Wow. When you put it that way, I guess not.”
We shook hands, I walked out, and didn’t bill them a dollar. I didn’t want to have my name associated with that project in any way, shape, or form. A few months later, the company went bankrupt, and I wasn’t the least bit surprised.
You’ve only got one reputation. When a client asks you to do work that you’d be ashamed to admit in public, it’s time to get everybody on the same page, make sure the client won’t change their mind despite your best sales pitch, and then gracefully step aside. They’ll find another consultant willing to do it – there’s always a few – and you’ll find another client.