Pop Quiz: What Do These Things Cost?

Posted on by Brent Posted in Blog Posts | 13 Comments
samsung-1tb-solid-state

The Price is Right

Arrange this list of things in order based on how much they cost:

  • SQL Server Standard Ed. licensing for four cores
  • SQL Server Enterprise Ed. licensing for one core
  • One typical day of employee time
  • One week of a team’s time (4 people)
  • 64GB of memory
  • 256GB of memory
  • A mirrored pair of 1TB SSDs

Without searching the web for prices, put those items in order. Do it on a scratch piece of paper – first estimate how much you think each one costs, then put them in order.

How does this impact the way you do performance tuning, and what solutions you try first?

For my answers, search for “The Answers” on this page, and read my comment.

Reading Critically: VMware Virtual SAN Performance with SQL Server

Posted on by Brent Posted in Blog Posts | 13 Comments

vmware-whitepaperSeriously, I’m not trying to pick on VMware documentation, but lately there’s some odd stuff coming out. Last week it was a really bad book, and today it’s a technical white paper from VMware itself – VMware Virtual SAN Performance with Microsoft SQL Server. It’s an easy read – a 9-page PDF – but here’s the highlights.

Think about the white paper’s title for a second: it sounds like it’s going to test how well the storage performs with SQL Server. Virtual SAN – so we’re testing a storage-intensive workload, right? Wrong, because the very first paragraph of the executive summary says:

“Experiments show the Virtual SAN storage sub-system is never the bottleneck and the workload saturates the host CPU while the I/O latency remains constant.”

Oooo, must be a lot of throughput if the CPU goes to 100%, right? Surely they didn’t handcuff the guests with small amounts of CPU power to keep the storage from being a problem. Let’s check out page 4:

“The entire DVD Store benchmark tools, including the query generator and the database backend, were encapsulated in a single virtual machine, which ran the Microsoft Windows Server 2008 R2 operating system and Microsoft SQL Server 2008. The virtual machine was configured with 4 virtual CPUs (vCPUs), and 4GB of memory.”

They ran the front end app and the database on a single VM with 4GB RAM. This isn’t a load test, it’s a grudge match! And it’s utterly pointless because each host has 128GB memory. They’re just leaving the memory sitting idle. Alright, so what kind of throughput were they able to get with these little toys?

“An aggregated “orders per minute” of 77,206 across 12 DVD Store instances”

Wait – what? I’ve never seen a SQL Server benchmark that just added up all of the transactions that a bunch of completely isolated SQL Servers ran, and called that a single score. You wouldn’t run a real store that way – that’s 12 different databases that can’t see each other.

But let’s play along anyway – 77,206 orders per minute divided by 12 databases equals a whopping 107 transactions per second. If you haven’t done benchmarking before, that number is what we in the performance industry call “low.” Even when combined, 1,287 transactions per second isn’t all that impressive – especially when I bet a single properly configured VM with that exact same hardware could eclipse all 12 of these misconfigured guests. (Not to mention the licensing overhead of the setup in this whitepaper.) To put this in perspective, Anandtech got 1,940 orders per second on a single server back in 2009.

I don’t have anything against VMware’s Virtual SAN product – I bet it’s awesome – but this white paper doesn’t do it justice.

When Consultants Should Fire their Client

Posted on by Brent Posted in Blog Posts | 6 Comments

First off, if you’re one of my clients, rest assured that this story isn’t about you – even if I’ve graciously terminated our relationship at some point in the past. This is an actual true story from a few years ago.

When I’m working with a new client, we check backups together first. We all need to be on the same page about how often the backups are performed, whether they’re succeeding or failing, and where they’re being written.

It’s not unusual for us to discover problems together. Backup jobs fail, custom scripts miss newly added databases, or mission-critical databases are in simple recovery mode. These things don’t freak me out – we’re all human, and we make mistakes, and after all, I’m not being called in because things are great.

But at one brand new client, things went differently. We discovered that their main production server – housing live financial transactions for their customers – was only backed up once per day, and the backups were written to the same drives that housed the SQL Server databases.

If they lost Windows, the RAID array, the RAID controller, or any number of things, those backups would be useless.

How That Conversation Went

Me: “Do you understand the risks here?”

IT Team: “Yes.”

Me: “Have you explained this risk to management?”

IT Team: “Yeah, but they don’t really get it, and it’s not a priority for them right now.”

Me: “OK, let’s go explain it to them together right now. I have a fun way of explaining things like this.”

IT Team: “Actually, we’d rather not. Let’s focus on the performance issues today.”

Consultants, Here’s Your Sign

Danger.

Danger.

The biggest risk for me as a consultant isn’t someone skipping on their bill.

The biggest risk is getting my name associated with a really, really bad idea. I only have one reputation, and I don’t want to be known as “the guy who stood by while Company X lost their entire database.”

Whenever you’re working on a project, don’t think of it as private – think of all of your peers and future customers watching over your shoulder. Can you defend what you’re about to do? If you were called to testify in court about the project, would you feel good about it?

Breaking Up is Easy to Do

Me: “I want to make sure I understand so we’re all on the same page. For you, getting the performance fixed is more important than having backups.”

IT Team: (after much hemming and hawing) “Yes.”

Me: “And I’m going to guess you wouldn’t let me put that in writing and get management to sign off on it. I know that sounds crazy, but I’m an outsider, and here’s the scenario: I could easily sit down at that production server, and while I’m working, someone else could drop a database. It’d be impossible for me to prove I wasn’t the one who dropped it, and it would cost me a fortune in court just to try to defend myself. So I’m even willing to work on this – but only as long as we whip something up in writing saying that the company executives understand the risk, and they indemnify me from all risk.”

IT Team: “Uh, no, we can’t do that.”

Me: “So if you were me, would you take on this risk?”

IT Team: “Wow. When you put it that way, I guess not.”

We shook hands, I walked out, and didn’t bill them a dollar. I didn’t want to have my name associated with that project in any way, shape, or form. A few months later, the company went bankrupt, and I wasn’t the least bit surprised.

You’ve only got one reputation. When a client asks you to do work that you’d be ashamed to admit in public, it’s time to get everybody on the same page, make sure the client won’t change their mind despite your best sales pitch, and then gracefully step aside. They’ll find another consultant willing to do it – there’s always a few – and you’ll find another client.

What I Wish I Knew Sooner as a DBA

Posted on by Brent Posted in Blog Posts | 7 Comments

Mike Walsh asked a few folks for 4 things they wish they’d have known earlier.

1. You have customers, not users or coworkers. Every person you work with today is a potential reference and a customer for you down the road. Treat them with professionalism and respect.

2. Focus on your customers’ pains. Ask them what sucks, and how you can relieve that pain. Your database server won’t give you a raise for decreasing fragmentation.

3. Keep it short and sweet. Typing a lot doesn’t show off your knowledge – it shows that you don’t respect others’ time. Give them the right information to solve their problems in as little time as possible.

4. Don’t help people for free via private emails. I used to spend hours answering questions for people who can’t be bothered to read the manual. Thing is, they don’t thank you, and they don’t respect your time. Help people in public, under bright lights, at places like Stack Overflow and DBA.StackExchange.com where your work will show up in the search results for the rest of time – thereby helping countless others who have the same problem. These days, when strangers email wanting free help, I use these email templates.