First things first, I don’t want to be your remote DBA. No thank you, unsubscribe. I never want to be on call again. From 1993 until 2008, I had an electronic tether – first in the hotel & restaurant industry, then IT. When I went to work for Quest Software as an evangelist in 2008, one of the biggest job benefits was being able to not answer the phone on weekends and holidays. Oh, sweet personal life, how I missed thee.
But I get tempting emails from companies that usually go like this:
“We’ve got a few database servers that are mission-critical. We don’t have enough work to keep a full time DBA busy, so we’d like to save money by hiring a part-time remote DBA. We just want them to remote in for several hours a week, make sure the trains are running on time, and then when things break, they can jump in and troubleshoot it.”
It all sounds so easy, right? Just set up the servers with a few good scripts, and you can just keep cashing checks. After all, DBAs don’t work that hard, right? (It always looks that way from the outside.)
Those last few words are the killer: “when things break, they can jump in and troubleshoot it.” Remember, the company’s database server is mission-critical, and when it’s down, they want someone quick, fast, and in a hurry.
That only works if the consultant isn’t tending to anyone else’s emergency.
So you end up with a few kinds of remote DBA firms:
Remote DBA Option 1: The Lone Gunman
A DBA quits his job and starts doing contracting full time. He wasn’t that busy at his old company anyway, so he makes them an offer they can’t refuse: he’ll keep the lights on for them, and he’ll remote in for emergencies. He’s desperate to get his first client in, so he lowballs them on a retainer and hourly fee. He’s now managing a few servers for a single client.
Over time, he adds more clients with more servers, but because his rates are really low, it takes him quite a while to build up enough momentum to keep the bills paid reliably. Emergencies don’t strike all that often, and he’s able to keep juggling these balls in the air to keep everybody happy. Clients love him because he’s way cheaper than a full time DBA, he’s smart, he knows their servers well, and he always answers the phone when there’s a problem.
Until that one day when he can’t.
Maybe he’s on the phone with someone else, or he’s at a ball game with a dead phone, or he’s on vacation out of the country, or his block has a power outage. He’s only human, and he’s only one human.
It doesn’t happen often, and some companies are okay taking that gamble.
Remote DBA Option 2: The Global Team
A company specializes in remote DBA work, and they hire a team of people. You can’t make the financial numbers work on this business with just senior people, because senior people want to get out of the on-call game. You have to pay senior people serious money if you want them to do on-call full time without burning out. Instead, they hire a few tiers of different skills from junior DBA to senior DBA. They set up on-call rotations, plus escalations so that if a junior person can’t solve a problem, they can bring in a senior person.
To make the juniors’ jobs easier, they build standardized runbooks that lay out formal procedures on how to handle a failed backup, a stopped SQL Server service, a drive that’s out of space. When the firm brings in a new client, they have a process for that too, vetting the server and getting the client to approve the runbook.
The remote DBA company’s entire business is based around standardizing processes so that any member of their DBA teams can step in and handle a call.
And that’s exactly what happens.
When you call a remote DBA firm, ideally you’ll get your primary contact – just as you get the Lone Gunman when you call him – but that’s not going to happen when the server breaks after hours, on weekends, or on holidays. The bigger the DBA firm – the more successful they get – the more likely you are to talk to someone who’s only seen your systems once or twice, if ever. It’s a qualified voice (hopefully), but it’s not a familiar one.
It’s also more expensive – because building this kind of around-the-clock support team, training them, managing them – it all costs money. This firm will be more expensive than the Lone Gunman, and I’ve seen cases where it’s been more expensive than just hiring a full time DBA. However, you could argue that it’s better than a full time DBA because it includes around-the-clock support at all times, even when your own Lone Gunman FTE is off on vacation.
Remote DBA Option 3: Cloud Services
The marketing behind cloud services – and I mean things like Windows Azure SQL Database and Amazon RDS, not virtual machines running plain old SQL Server – says that the cloud vendor does the lights-on support for you.
They do this by simply controlling the platform entirely. You don’t get the option to remote desktop into your WASD instance and apply patches. You don’t get involved in HA or DR failovers. When things break, the vendor fixes them.
In theory, it’s way cheaper and much more flexible. You solve multiple problems at once – not only did you have too little work to keep a full time DBA busy, but honestly, you probably had too little work to keep a properly configured SQL Server busy full time, too. With well-designed resource sharing and the economies of scale, you can get the right amount of database power and people that you need.
Except today, companies don’t want to move their databases to the cloud just to solve the part-time-remote-DBA problem. Today, they’re still too focused on keeping the rest of their infrastructure in-house, so moving everything to the cloud just because they can’t keep a DBA busy seems bass-ackwards. Instead, they compare the pros and cons of the Lone Gunman, the Global Team, and just going without DBA attention altogether.
As a DBA in the 2000s, I was worried that outsourcing & remote DBA firms would take over conventional DBA jobs. They didn’t. In the early 2010s, DBAs were similarly concerned about cloud services taking over DBA jobs. That hasn’t happened yet either, and based on what I’m hearing from companies, I don’t think it’s going to happen in the near term (3-5 years).
Today’s Options are Bad, but the Problem Still Exists
I know a lot of my clients would love to get this problem solved – but I just don’t have a good answer for them. Relational databases aren’t getting easier to manage. In fact, with every new release of SQL Server, I see new ways for customers to shoot themselves in the foot with ever more powerful weaponry. (Don’t get me wrong – I love new features, but they don’t manage or configure themselves.)
Part of tomorrow’s answer will be software. For companies with SQL Server and sysadmins, I like where Idera’s going with SQL Elements. For shops where the developers manage SQL Server, I love Stack Exchange’s open source Opserver. Both tools still require meat popsicles to interpret the dashboard and take action, but they’re bringing insight to users without formal DBA training.
I love watching this market as it grows and changes over time. I don’t have a horse in this race – I’m never going on call again, and we don’t want to hire employees to put them on call – but I get excited every time I see a new solution.
Because I don’t want you to be on call either.