I wasn’t originally going to share this, but given how PASS is handling the things that you do know about, it’s time for you to know the even worse story.
In this June’s NomCom election, when you signed in to vote, your PASS login credentials were shared with a third party. I tweeted it at first because I was so shell-shocked – this just couldn’t possibly be happening, and it had to be a misinterpretation on my part.
After that initial discovery, we took the vulnerability reporting privately to PASS. Volunteer Andy Warren (@SQLAndy – SQLAndy.com) led the charge, working with PASS to explain the problem, and reporting back to other security-focused community members to report on the status. It took months, and on the bright side, it was fixed in time for the current election.
I just kept waiting for PASS to do the right thing. I’m done waiting.
Here’s what should have happened:
- All PASS passwords should have been reset.
- The community should have been told that your password was compromised, and that you’d have to log into the PASS web site to change it.
- That would have taken care of the “update your profile” voting eligibility issue.
But instead, you’re just now hearing the news from me.
Go change your SQLpass password now. As a data professional, you should also be using a unique-per-site password. I use 1Password, and I also hear good things about Lastpass.
If you think we’re not giving PASS feedback privately, and if you think PASS acts appropriately on it, here’s your sign. And it is one ugly, dire sign.
Update 9:13AM Eastern: It Gets Worse
Now we’ve found out that PASS doesn’t understand the vulnerability issue of entering your username and password on a different web site that they don’t control. Adam Jorgensen’s comment below says “At no time were passwords at risk” – which means the entire Board of Directors did not see the security vulnerability even when told of the issue, and didn’t contact us for any clarifications. Now we’re at serious levels of negligence, and I just don’t know how to make the problem more clear.
You can learn more about man in the middle attacks here, but here’s a quick intro:
- PASS Member goes to VotingWebSite.com
- VotingWebSite.com says, “Hey, what’s your PASS username and password so I can let you in to vote?”
- Member puts it into the VotingWebSite.com form
- VotingWebSite.com gets the data, and sends it to SQLpass.org asking, “Is this a correct login?”
- SQLpass.org says yes or no
Game over. A third party has your login and password.
Update 10:30AM Eastern: It Gets Even Worse – All Passwords Exposed
George Stocker (@gortok – GeorgeStocker.com) noticed that all PASS member passwords are actually vulnerable. I recorded a video to repro the steps, and we’ve sent the details to PASS.
At this point, if you use the same password at PASS that you use ANYWHERE ELSE, CHANGE YOUR PASSWORDS AT ALL OTHER LOCATIONS IMMEDIATELY. This vulnerability has possibly been in place for years. We’ll report back with more information after PASS fixes the vulnerability.
Update Sun, Sept 28: Vulnerabilities Repaired
Whew. You’re safe. Here’s where to learn more:
83 Comments. Leave new
Seriously? PASS BoD, is this true?
And some folks wonder why I wrote that SSC editorial saying we don’t care about security.
If this is true, and data professionals let this happen, then my point has been reinforced.
KBK – you should probably link to the post to drive it home.
Link: http://www.sqlservercentral.com/articles/Editorial/114099/
Whoa. Glad I don’t use this one anywhere else.
I’d invalidate the election right now.
Ooo, I actually didn’t think of it as a vulnerability in the election results. To some extent, I doubt that actually happened because people who cared enough to vote in NomCom probably cared enough to vote in the BoD election, so they would have noticed immediately if there were widespread attempts to vote twice.
However, if you were looking for the passwords SQL Server professionals, I sincerely doubt your goal would be to change election results. There are much, much more valuable uses for the passwords of data professionals, and that’s what really scares me here.
If PASS didn’t believe it was a real vulnerability (and Adam’s comment indicates that’s the case), then they should have at the very least posted my question publicly and said, “Is this actually a vulnerability or no?” Or at least consulted security professionals like KBK or Denny. Anybody familiar with MITM attacks would identify the problem instantly. (Of COURSE the vendor’s not going to say there’s a problem.)
To be clear about why I’m so flippant about the vendor – the vendor actually offers MITM as a service.
This is not true. Pass uses a third party to handle the balloting. We use their recommendations (based on their leadership in that field) on how to connect with them. The nomcom election used a secure method based on those recommendations. Based on feedback from the nomcom we upgraded our approach to exceed their recommendations. We heard and acted quickly on the private feedback and this was communicated to the nomcom and board. At no time were passwords at risk and there is zero evidence of any improper account access or activity. Since no system is 100% secure we always recommend that PASS members change their password regularly and use that opportunity to update their profile. I would ask that before posting something this alarming that you validate the information you’re posting and do so with the right people. You can always reach me and I can clear things up. I’m always happy to do so Brent.
Adam – that’s simply not true.
Balloting was done on SimplyVoting’s systems. SimplyVoting hosted the username and password form. When users authenticated, SimplyVoting sent the username and password to PASS’s systems, and PASS validated it. This means SimplyVoting had the username and passwords during the authentication process. This is known as a man-in-the-middle phishing attack.
I understand that PASS didn’t proactively *send* the passwords to SimplyVoting – they didn’t have to in order to be vulnerable.
Adam – for more proof, here’s the vendor’s list of authentication options:
http://www.simplyvoting.com/website.php?mode=features#auth
The only safe way that doesn’t give Simply Voting the user passwords is the last one in the list. With ALL of the other methods, Simply Voting gets the user’s name and password during the login process. When you say the security was upgraded after the NomCom election, that’s true – at THAT point, and only that point, security was upgraded to the bottom one, and Simply Voting didn’t get the passwords. Prior to that, passwords were exposed.
Brent, let me get this straight passwords were not SENT to SimplyVoting, But SimplyVoting prompted for the password, and they authenticated against the PASS system. Who knows what SimplyVoting actually does with the password after they authenticate it. I guess I am safe since I don’t GET to vote and have never been to SimplyVoting.
Will – correct. As I understand it, PASS did not proactively send *ALL* username/passwords to SimplyVoting – SimplyVoting just got the usernames/passwords as users logged in to vote, and only if they logged in successfully. Classic man-in-the-middle phishing vulnerability.
Adam,
I’ll say it here again. If you are representing PASS, go respond on the official site and link here. It’s inappropriate for you to respond here as a board member.
Adam,
what you have said confirms that passwords were disclosed. If someone voted for the NomCom, then their password was seen by your third party. Not only that, but the validity of that password was confirmed when they checked back with PASS unless you implemented your own single sign-on solution.
As a security professional, I understand why companies do that with respect to cloud solutions authenticating back to LDAP (AD), for instance. However, in those cases, you typically see requests for attestations, etc. So here’s a very, very important question: What did PASS ask for with regards to those typical types of attestations. Given your role at PW, you should know exactly what I’m talking about.
Also, how would you detect if there has been improper activity or access? The third party knows the email and the password. Unless you’ve got some protocol like FB which checks the location where a log-in comes from, and someone is actively reviewing that or there’s an auto-block in place, you wouldn’t know. Just because a system logs doesn’t mean someone reviews. Do you have someone reviewing? If you don’t have one or both, you can say there’s zero evidence, but there’s no validity to your statement. This isn’t a personal attack, but simply a statement of fact from a security architecture perspective.
Found that SimplyVoting has this:
SSAE 16 SOC 1 Type I certified
Now, did you guys ask for the actual attestation itself? The attestation is effectively an opinion and two companies may issue that attestation with very different standards. So it’s important to review the attestation itself. Did anyone do that?
WHAT… THE… F$@@?
Stuart – yeah.
wow!
I just changed my password.
Thank you for sharing this, Brent.
Chuck – you’re welcome. And now I’ll go put on my flameproof pants.
Might want a flack jacket too because I’m sure you’ll take a lot of it.
However, let’s all take a step back and deal rationally with this like we would if we noticed a problem in our workplace. The issue has been identified. Now we need to arrange a solution, test the solution and move on.
Adam Jorgensen, now is not the time to go on the defensive. Show good leadership by ensuring the issues that have been brought to your attention are successfully dealt with. I believe that more than enough evidence has been provided to show that a possible risk exists.
Wow. Changed my SQLPASS password right after I read this post. I can second the referral for LastPass. I started using it a month or two ago after growing tired of trying to remember all the passwords in my scheme.
Ron – yeah, these password utilities are crazy helpful. Should be part of mandatory training for all IT folks.
Complete violation of trust. Should be criminal not to disclose this. Whom was the third party? What is their security and privacy policies? I am a PASS member by paying a lot of money to attend the Summit years ago therefore it was a client consumer transaction. I am sure lots are vulnerable from password reuse from this as well as the potential for voter fraud. This group who has all the id’s and passwords could log in an away the vote. Thanks for sharing and letting the public know. We are glad to know someone is looking out for the community.
So let me get this straight; the Professional Association for SQL Server’s email policy for SQLSaturday’s claims they can’t give chapters access to the email information of attendees, but they’re perfectly OK with Man-in-the-middle? And, NOT informing their membership about a potential compromise?
I think I’m just about done…
Stuart – correct.
I’m at a loss for words for anything other than “correct.”
Tim – the third party was SimplyVoting. Their privacy policy is here:
http://www.simplyvoting.com/website.php?mode=privacy
And it correctly states things like:
“We collect personally identifiable information like name, email address, and financial information such as credit card number, when submitted by account owners for management of their ballots.”
“All elector authentication credentials (elector ID, password), vote information and vote receipts are encrypted via Secure Socket Layer as well.”
(Which doesn’t matter, the point is that they get both your username and password)
“When a remote authentication method is used (web service, LDAP) the elector credentials are then transmitted to the remote authentication server and will not be stored on our servers.”
Fingers crossed that they’re truthful – but why cross fingers on something as important as data professional passwords?
Couple notes, since I was involved in part of this:
– The team at Simply Voting was open and helpful when I contacted them (I did do without mentioning PASS, but they connected the dots) and explained they provide a variety of options for authentication to customers, some who are move savvy than others.
– To the extent that they have the option we’re discussing here, they take security pretty seriously – look at the stuff they have on their site about it
– It was tough going to get PASS to understand the issue, but finally they did and made the change to do it the right way this year. It should have been easier, and I’ll argue it shouldn’t have happened to start with
– I rate the risk of password compromise low. Not zero, but low.
– I debated writing about it when I knew the problem and after it was fixed, and elected not to. As you might imagine I’m second guessing that decision today, it’s not up to me whether you trust Simply Voting or not. My goal was to fix the problem going forward and not cause damage to PASS if it could be helped. I also hoped that PAS would tell the story, because while it was a fail, there was (or could have been) a win there too.
I don’t believe the problem compromises this election (because the issue was fixed), and I don’t see any sign that the NomCom election was compromised. The worst worst case would be someone compromised SV, got the list, picked an account that had an updated profile (or updated it), and then voted before the “real” voter voted. Not impossible, but not likely.
I’m not a PASS defender on this, their handling of it infuriated me, but this isn’t a problem unique to PASS and I hope we can keep that in perspective while taking steps to further sensitize the full time staff and the BoD about how serious security is.
Andy
Andy – thanks for the comment, but I’ll disagree on one thing – that is definitely not the worst worst case scenario.
The worst case scenario is that a PASS member uses the same username/password for both their company and SQLpass.org, and their company got compromised.
If PASS was monitoring account logins, it truly wouldn’t matter – they’d never see the logins.
For all we know, PASS’s password sharing could be at the root of any number of recent corporate penetrations.
When you’re sharing passwords with other sites; even if you believe they’re trustworthy, you still have to worry about any vulnerability that company has.
Remember the OpenSSL fiasco? Sure SimplyVoting sends those passwords through SSL; but were they using OpenSSL at the time? Were they patched when this happened? I’m an outsider, so I’m not too sure of the timeframe for this.
SimplyVoting has a nice juicy target painted on their back: Attackers know they accept 6 forms of authentication that enable them to get a user’s username and password. Even if they employ the best security professionals in the world, they simply can’t move faster than a zero-day hacker.
they’re. My mistake is now immortalized for all eternity. Damn you, autocorrect.
What mistake? I don’t see any mistakes.
Sincerely,
Manuel Correcto
George – yep, extremely well said.
Agree with Brent. The real issue is password re-use. A lot of professionals know they shouldn’t, but they do it anyway. Company, bank account, etc.
And again, it’s not about a security flaw; it’s about how it was NOT handled. If the BoD knew about this and didn’t inform membership about a potential security flaw, do we continue to excuse the board’s action as “good people with good intentions gone bad”? When does it become an issue of integrity?
Just… just… no comment.
Andy L – and by doing so, you left a comment. You fell victim to my diabolical plan.
How could database professionals look at that list of Simply Voting options and not see the vulnerabilities? PASS should have created the single sign on process to authenticate users and updated a member ballot table to avoid the potential for duplicate votes. Good call, Brent
Karen – agreed. When I saw that list of options back in June and we alerted PASS, I was sure it was just a simple oversight by an overworked volunteer not understanding the security risk. If the issue was brought to the entire Board of Directors, and they didn’t understand the vulnerability, *and* they didn’t seek clarification from a security professional, that’s downright negligence.
Karen,
see my editorial from SSC (linked near the top). It shouldn’t be asked of just data professionals, but of IT professionals in general. Security isn’t given much thought. That’s how these things happen.
Can’t argue with that, I didnt say it well/correctly!
This just gets crazier. A group dedicated to databases cannot handle basic security issues.
Thanks Brent for sharing this. I just changed my password using Password Safe.
Dave – you’re welcome. I’m glad I could help make a difference.
Along the same lines, PASS doesn’t appear use HTTPS when you use your credit card to pay for registration.
Steven – hmm, are you sure? I just tried to register and got https by default, but I didn’t go through to the credit card step. Can you upload a screenshot somewhere or email it to me?
OK, I just tried it and HTTPS does show up when I clicked on registration just now.
However, when I did the registration originally, back in late 2013, there was no HTTPS present.
Steven – yeah, the events registration with credit card is protected. It’s just the account creation and password changes that are vulnerable. I added a video in the post above to illustrate it.
(redacted)
Should the voting website not have collected the users vote then redirected to SQL Pass website for authentication to take place, then once authenticated send a token back saying AUTHENTICATED then record the vote similar to buying through an e-commerce website paying via PayPal which lets the user add things to your basket then when you click checkout it says you will be redirected to authenticate with PayPal then once the payment is confirmed the order gets placed? No need for any intermediary to hold a copy of the credentials.
It would have been simpler to generate a one-time password (OTP) and send out via email. Log on to the site with the OTP, vote, done.
True 🙂
(redacted)
Charles – you’re absolutely correct. I edited the comment to remove it because we’re worried about the zero-day vulnerability in the PASS site. We’ve contacted PASS and raised the issue as extremely urgent. You get a gold star for noticing it too!
Ahh, oops shouldn’t have written that hope it doesn’t have bash too although if it does better delete this post too
They’re running DNN on Windows, so ironically, they’re probably okay. (Although same guidance goes for load balancers, routers, cache appliances, etc.)
Thanks for sharing this Brent. It’s a big WTF, but I do appreciate transparency.
Tim – you’re welcome, sir.
So wait … until the (redacted) thing is fixed, how much good will changing the PASS password do?
Seth – you can’t just change your PASS password. You have to consider anywhere that you used the same password, and change it everywhere else.
Right, I get that. I’m just wondering about the efficacy of PASS-specific password changes.
Seth – originally, I only knew about the elections vulnerability, and PASS had fixed that one – but not alerted the public. This blog post was written solely to discuss that original vulnerability, and if that was the only one, then PASS-specific password changes would have been fine.
Unfortunately (or fortunately?), as people read this post, they started poking around in wider PASS security issues, and that’s when George found the additional vulnerabilities that make PASS-specific password changes useless.
Over at VMware headquarters, the whitepaper guys are probably saying, “phew, at least it isn’t our turn in the crosshairs this week!”
Nic – HAHAHA, true!
Unfortunately now I am faced with the very real problem of coming up with something witty or trenchant for me new PASS-word.
…since everybody’s gonna see it, and all.
Wow.
My own anecdote: I used Google plus addressing when I signed up with PASS, so they are the only ones who had that particular address. A few months ago, I started receiving spam at the PASS-specific email address. Way to go, people.
[…] Brent blogged about PASS’s lack of security surrounding their voting system. In essence, it worked like […]
I’ve posted a followup to this discussion (specifically regarding my disclosure of the vulnerability) on my blog.
In short, mea culpa. In longer form, just go ahead and change any site credentials that you share with SQLPass. Don’t worry about changing them on PASS until they fix this vulnerability, just make sure none of your other passwords or credentials match what you use for PASS.
George, I tried to post a comment on your blog but ironically I can’t remember my WordPress password. This was really the last straw for me. I do use unique strong passwords for the important stuff, but I really need to evaluate and adopt 1Password or LastPass this weekend. I’ve had several false starts over the years with KeePass, but I think now that we’ve all got smart phones and cloud storage this 1 password per site thing can really work. Time to two-factor all the things.
Did PASS change their name to remove “SQL Server” or “Professional” from the name?
ZING
And by that I mean, Noel, that’s not very professional of you.
Yes, well “professional” is not part of my name. I guess I could change my name to ProfessioNoel, but then I’d have to raise my standards… and it would just be silly.
Just waiting to hear that Bobby Tables WAS able to log in and vote at PASS…
Shocking and saddening yet becoming less and less surprising as more behaviors are being made public.
But on the plus side: *silence*
Lucky me. My login/password for the SQLPASS website is not used anywhere else for anything else.
So the combination is… one, two, three, four, five? That’s the stupidest combination I’ve ever heard in my life! That’s the kind of thing an idiot would have on his luggage!
Well I am at least happy that I can tell my wife and kids that paranoia pays off for once. I don’t use the same combination of user/passwords but as I am getting older it is time to look into one of those new fangled utilities.
Sad thread to be reading as one of the members of this group.
Lucky me. I just went and changed my passwords at other sites but it was only reused at some older unimportant sites (Car forums). Any thing important has 2 factor auth where available or 16-32 character random passwords stored in a password vault.
Part of me is surprised that an organization such as PASS would have such a vulnerability but then I see the credit card leaks in the past couple of months and realize nothing is immune any more. Just make sure your up to date on your shots.
Please tell me this isn’t true and this is a well thought of marketing campaign you are running Brent?
If not, there is something that PASS should be aware of, and should be held accountable for. It appears to me that in that case they are having some major issues on how to deal with PII data, which seriously makes me frown since I would expect more from a group that considers themselves a Professional Association, with board members that have knowledge and talent knowing how to understand with PII data. If this would be any retailer (Home Depot, Target etc…) to name a few, they would have to disclose that information, send out clear communication to its members and provide coverage in terms of fraud protection. So if any of the PASS BoDs is reading this, I would like to have a clarification on what happened, how data is stored, and how both PII data as well as password data has been transferred to another company. If this was done unencrypted (passwords sent in plain text), I would consider that to be a rather severe data issue.
Dandy – yep, PASS has verified it and fixed the vulnerabilities (although not emailed the users yet). More details here: http://www.brentozar.com/archive/2014/09/change-sqlpass-password-right-now/
[…] The Bigger #PASSVOTES Problem: Your Password Was Shared by Brent Ozar […]
[…] of weeks about the Professional Association for SQL Server wasn’t just about the election or the password controversy, but about the decision to become simply PASS in all marketing materials (gonna need a new hashtag […]
[…] https://ozar.me/2014/09/bigger-passvotes-problem-password-shared/ […]
[…] https://ozar.me/2014/09/bigger-passvotes-problem-password-shared/ […]
[…] Hunt et Brent Ozar m’ont définitivement convaincu d’utiliser 1Password. Maintenant chacun des sites et […]
[…] could be compromised via a man in the middle attack. That was corrected after the election, but as Brent Ozar noted in a blog post PASS did not warn affected users. That led to the discovery that passwords […]