Sooner or later, you’re going to get a new client with a request that makes you a little nervous. Here are a few examples I’ve seen over the years:
- “We have a multi-terabyte mission-critical database with no backups, and we want it to go faster.”
- “We have a 15 year old system, long out of hardware and software support, and it’s never been patched.”
- “We’re storing personally identifiable data about minors without any permission or encryption.”
When my admin-senses go off, I ask myself:
What’s the worst that could happen if this system goes down? Will people die or be injured? There are computer systems that control hospitals, life support, pharmacies, nuclear power plants, etc. You wanna know doggone well what’s happening on the other end of the wire. Will people lose jobs? Lose money?
Once it starts going down, can you stop it? One of the best answers here is, “I can fail over to another system where there’s a known-good backup, and the well-documented failover process has been tested repeatedly and recently.” The ugliest answer is, “We don’t have backups, and we don’t have any other systems capable of sustaining our production workloads.” In the latter situation, once things start going south, you’re simply screwed. You may not have the authority or capability to bring up a replacement system.
Could you make a mistake that brings the system down? I’m not asking how likely it is – I’m just asking if it’s even possible. Can you conceive of ways you could accidentally knock it down? If that chance is even remotely possible – and it’s always possible – then you need to think about how you could communicate those risks to the client, and how you could reduce those risks.
Could you afford to defend yourself? Think about the size of the company on the other side, the number of affected end users, and the severity of the incident. How much are they going to invest in a legal effort to come after you? Even if you didn’t cause an outage, could they make your life miserable in court?
What business am I giving up by taking this? If you don’t have anything else going on, you’ll be more tempted to take risky work. That isn’t wrong, necessarily, but you just want to be honest with yourself about why you’re doing it. When your plate becomes more full, you need to make different decisions.
Given all the risk, is the reward worth it? Sometimes folks will say, “If you just fix this one really bad problem on a completely antiquated and un-maintained mission critical system, you could earn a lot of business with us. We’re sure it will only take you an hour or two.” They’re not going to suddenly start investing the right amount of money in their IT systems – as soon as this emergency is over, they’re going right back to getting by on a shoestring.
Sometimes the reward is worth it – but sometimes, you just need to politely decline.