T-SQL Tuesday #36 (#tsql2sday)– Post-PASS Summit Depression

I had an email thread going with a prominent member of the SQL Server community today, where he confessed that he didn’t attend any sessions during the PASS Summit last week. He spent all of this time networking and catching up with people.

I, personally, can relate.

This year’s Summit was another incarnation of that ritual of SQL Server professionals meeting to share their knowledge, experience, and just have a wonderful time while doing so.

It’s been a few days after the Summit is over, and I’m definitely dealing with withdrawal.

My name is Argenis, and I’m a #SQLFamilyHolic.





(This post is part of the T-SQL Tuesday series, a monthly series of blog posts from members of the SQL Server community – this month, Chris Yates is hosting)

DBA Best Practices: A Blog Series



After the success of the “Demystifying DBA Best Practices” Pre-Conference that my good friend Robert Davis, a.k.a. SQLSoldier [Blog|Twitter] and I delivered at multiple events, including the PASS Summit 2012, I have decided to blog about some of the topics discussed at the Pre-Con. My thanks go to Robert for agreeing to share this content with the larger SQL Server community.

This will be a rather lengthy blog series – and as in the Pre-Con, I expect a lot of interaction and feedback. Make sure you throw in your two cents in the comments section of every blog post.

First topic that I’ll be discussing in this blog series: The thing of utmost importance for any Database Administrator: the data. Let’s discuss the importance of backups and a solid restore strategy.

Care to share your thoughts on this subject in the comments section below?

Database Mail and SMO are indeed supported on 64-bit, Standard Edition instances of SQL Server 2012


This is something that comes up rather regularly at forums, so I decided to create a quick post to make sure that folks out there can feel better about SQL Server 2012.

If you read this Web article, “Features Supported By Editions of SQL Server 2012” as of time of writing this post, you will see that the article points out that these two features are not supported on x64 Standard Edition. This is NOT correct. It is most definitely a documentation bug – one that unfortunately has caused some customers to sit on a waiting pattern before upgrading to SQL Server 2012.

Database Mail and SMO indeed work and are fully supported on SQL Server 2012 Standard Edition x64 instances. These features work as they should.

I have contacted the documentation teams internally to make sure that this is reflected on next releases of said Web article.

Ghost Records, Backups, and Database Compression…With a Pinch of Security Considerations

Today Jeffrey Langdon (@jlangdon) posed on #SQLHelp the following questions:


So I set to answer his question, and I said to myself: “Hey, I haven’t blogged in a while, how about I blog about this particular topic?”. Thus, this post was born.

(If you have never heard of Ghost Records and/or the Ghost Cleanup Task, go see this blog post by Paul Randal)

1) Do ghost records get copied over in a backup?

If you guessed yes, you guessed right. The backup process in SQL Server takes all data as it is on disk – it doesn’t crack the pages open to selectively pick which slots have actual data and which ones do not. The whole page is backed up, regardless of its contents.

Even if ghost cleanup has run and processed the ghost records, the slots are not overwritten immediately, but rather until another DML operation comes along and uses them.

As a matter of fact, all of the allocated space for a database will be included in a full backup.

So, this poses a bit of a security/compliance problem for some of you DBA folk: if you want to take a full backup of a database after you’ve purged sensitive data, you should rebuild all of your indexes (with FILLFACTOR set to 100%). But the empty space on your data file(s) might still contain sensitive data! A SHRINKFILE might help get rid of that (not so) empty space, but that might not be the end of your troubles. You might _STILL_ have (not so) empty space on your files!

One approach that you can follow is to export all of the data on your database to another SQL Server instance that does NOT have Instant File Initialization enabled. This can be a tedious and time-consuming process, though. So you have to weigh in your options and see what makes sense for you. Snapshot Replication is another idea that comes to mind.

2) Does Compression get rid of ghost records (2008)?

The answer to this is no. The Ghost Records/Ghost Cleanup Task mechanism is alive and well on compressed tables and indexes. You can prove this running a simple script:


USE GhostRecordsTest

                      myWideColumn varchar(1000) NOT NULL DEFAULT ‘Default string value’)

GO 10

DELETE myTable WHERE myPrimaryKey % 2 = 0



TraceFlag 2514 will make DBCC CHECKTABLE give you an extra tidbit of information on its output. For the above script: “Ghost Record count = 5”

Until next time,


T-SQL Tuesday #028 – Jack of All Trades, Master of None?


I’ve lost count of how many times I’ve heard that phrase.

Are you specialized? On something? Or anything at all? Has that been a good or a bad thing? Why?

Are you the SQL guy at work? Or the one who does everything?

Do you code? And configure wireless routers at work also?

If you had to pick one thing to specialize on, what would it be?

Over the course of my career I’ve worn many many hats. I always felt I was doing fine, had a stable job, but wasn’t quite fond of my prospects for the future. Then a friend said that I should focus on one thing and be the best at it. And while I’m most certainly NOT the best at it, I’ve gotten progressively better on it, to the degree that I’ve been called an ‘Expert’ by some (hate that word!) – I’d rather be called ‘knowledgeable’. My career took off like a rocket after I specialized, and certainly choosing to focus on one thing (SQL Server, in my case) has been one of the best decisions I’ve ever made. I’ve also been careful of not forgetting my roots as a SysAdmin – and always try to keep up with changes on the Windows/SAN/Networking front, but not with the same level of intensity.

So, in this installment of T-SQL Tuesday I’d like to ask you to blog about your experience. Tell us why you specialized, or why you’d like to specialize. If you don’t think that specialization is a good thing, tell us why. Discuss. Argue your point(s).


  • Your post must be published between 00:00 GMT Tuesday March 13th, 2011, and 00:00 GMT Wednesday March14th, 2011
  • Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
  • Trackbacks should work, but if you don’t see one please link to your post in the comments section below so everyone can see your work

Optional, but highly encouraged…

  • Include a reference to T-SQL Tuesday in the title of your post
  • Tweet about your post using the hash tag #TSQL2sDay
  • Consider hosting T-SQL Tuesday yourself. Adam Machanic keeps the list.

A Few Thoughts on the SQL Server MCM Program

I took the MCM lab a couple of weeks ago and passed. I still don’t believe it – it’s been quite the journey.

There have been many decisive moments in my career; one of those was back in 2010 when I decided I would work on obtaining the certification after Brent Ozar blogged rather extensively about his experience with the program. I progressively worked on aspects of SQL Server that were out of my comfort zone – and that paid off. But simply put, I have the cert – but still don’t consider myself a true Master. And I probably never will.

I might be okay in certain areas of SQL, maybe even good in others – but I don’t master them all. I intend on getting better on those areas, but that won’t happen overnight.

SQL Server is a mammoth of a product. It’s actually a suite of products – and the MCM certification focuses on only one aspect of it: the Database Engine. No SSAS, no SSIS, no SSRS.

So you might be wondering: Is it worth it then? Why work so hard on obtaining a certification that calls you “Master” when it doesn’t test your skills on every single feature of SQL Server?

To me, it was. And then some.

The fact that I value most after going through the program? Knowing exactly what I don’t know. In other words, looking at a problem and knowing that I won’t be able to solve it right away, yet still I will have an idea of where to begin digging for information to conquer it.

You see, when I decided to go for the MCM I would have probably ranked myself a 6/10. So I went and trained myself on those areas where I felt I wasn’t strong enough. I read blogs, I watched videos, I attended SQLSkills immersion events (lucky me!), I went to SQLSaturdays, and even a couple of PASS Summits – but all of that was theory, not a whole lot of practice. And when you get to the MCM Lab, you better had practiced. You better had familiarized with that particular feature you’re being challenged with – at least a tiny bit. Learning from BOL on the fly during the MCM Lab is probably just going to be a waste of time – trust me on that one.

The other thing I learned along the way? The power of networking. If you are a SQL Server professional, in any function – DBA, Dev or what have you – allow yourself a couple of minutes to brag about what you do: our community is awesome. Better than anything I’ve seen out there. I haven’t met a single person in the SQL community that’s not willing to help a colleague.

And that’s precisely what helped me prepare the most for the MCM: helping others. Perhaps that will help you get there as well?



On the Topic of Lost SA Passwords on SQL Server 2000…

Since it looks like everything I blog about lately is showing how to get onto SQL instances to which you don’t have formal credentials, I figured what the heck – let’s do a post on SQL 2000.

Earlier on today Saurabh Sapra [twitter] sent a tweet to SQL Server MVP Thomas LaRock [blog|twitter]:


To which Tom replied:


I was flattered. But I had no posts on that subject. So in turn, I replied:



It turns out that it is straightforward. And I will show you how to do just that – recover the SA password of a SQL 2000 instance. It’s easier than you think. As a matter of fact, I’m going to show you how to recover the password for any SQL login that has a current session established against a SQL 2000 instance.

Caveat – and this is important – you cannot recover the password if the user does not have a session established. I.E., if it is not logged on to SQL. I haven’t heard of a way to recover an SA password if there isn’t a connection established to SQL. If you know how to do this, please drop me a line.

Let’s dive right into it.

On Enterprise Manager, I registered the local instance using the SA login – then dropped the BUILTIN\Administrators login.


Nothing shocking there – just showing you that there is at least one active connection to SQL (Enterprise Manager itself) and that there are no other SQL logins on the box.

Next, I opened up a CMD window, and created a memory dump of the sqlservr.exe process. I can do this because I am a local Windows Administrator on this box. To achieve this I used the SysInternals’ ProcDump utility. Then I took the memory dump and parsed it through strings.exe, another SysInternals utility, essentially filtering from the memory dump anything that doesn’t look like a string.


Now I opened dump.txt in Notepad, and searched for the host name of my SQL 2000 box immediately followed by sa. In this case my VM is named “HURRICANE”, so I searched for “HURRICANEsa”


And there you have it. The password is found between the string “HURRICANEsa” and the string “MS SQLEM(LOCAL)ODBC”. The SA password for this instance is “[email protected]!”.

Obviously your memory dump will look different than mine, given that your SQL client string will probably be different than the one Enterprise Manager uses – which I leveraged for this example. Also, my SQL 2000 instance had just been restarted, so I had less ‘stuff’ in memory than you probably will. You might even get a false positive. My point: YMMV.

As I mentioned above, this technique also works for any SQL login with an established connection, regardless of its privileges on the server. Moreover, this technique also works on SQL Server 2005 and (I believe – haven’t tested it) some builds of 2008 and 2008 R2. It wasn’t too long ago that the SQL team at Microsoft started to finally clear this area of memory after a session’s credentials have been verified.

Hope this was useful.



UPDATE: A couple of folks asked me which build I used to test this. I originally tested it on SP4a, and later confirmed with a more recent build, 8.00.2187 (but not the latest one that’s available only to customers with support agreements).

Leveraging Service SIDs to Logon to SQL Server 2012+ Instances with Sysadmin Privileges

Edit: I have confirmed that this is still valid for the all versions of SQL Server 2012 and above. As of this last update (Oct 2019), this works even on the yet-to-be-released SQL Server 2019. The binary paths may change across versions, but the idea is the same.

If you recall one of my previous blog posts, titled Think Your Windows Administrators Don’t Have Access to SQL Server 2008 by Default? Think Again I exploited the fact that NT AUTHORITY\SYSTEM was granted membership to the sysadmin server role by setup in SQL Server 2008 R2 and below to gain access to a SQL instance to which I had no access, since as Administrator on the box I could launch a cmd session as NT AUTHORITY\SYSTEM with Sysinternals’ psexec utility.

My friend and SQL Server MVP Jorge Segarra [blog|twitter] correctly pointed out shortly after that NT AUTHORITY\SYSTEM is not a member of the sysadmin server role in SQL Server 2012, codename Denali. And as of Release Candidate 0 for this version, this holds true.

What also holds true as of RC0 is that the Service SID for a number of services (at least three, the SQL Engine itself, SQL VSS Writer and Winmgmt) are members of the sysadmin role. And so in this post I’d like to demonstrate that it is possible to exploit one of these services’ level of access to hop onto a 2012 (or 2014) instance as sysadmin.

The target: a named SQL instance called “DENALI_RC0” on one of my desktop PCs. Having dropped my login on SQL, when I try to logon to the instance I get the usual message:


I picked a service to become “the victim”. The SQL VSS Writer service seemed to be a good candidate: innocuous enough. If I stop it and restart it, no big deal.

I launched regedit and browsed to HKLM\SYSTEM\CurrentControlSet\services\SQLWriter – this is what I saw:


Now being an Administrator of this PC as I am, I went ahead and renamed sqlwriter.exe to sqlwriter.exe.orig, and put a copy of SQLCMD.EXE on C:\Program Files\Microsoft SQL Server\90\Shared.

Then I renamed SQLCMD.EXE to sqlwriter.exe.

Obviously kicking off the SQL VSS Writer service was not going to do anything – just error out:


So I replaced the ImagePath for sqlwriter on the registry with this:

“C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe” -S CSHQAFERNANDEZD\DENALI_RC0 -E -Q “CREATE LOGIN [CORP\Argenis.Fernandez] FROM WINDOWS; EXECUTE sp_addsrvrolemember @loginame = ‘CORP\Argenis.Fernandez’, @rolename = ‘sysadmin'”

And now I kick off the sqlwriter service again, expecting it to error out…but with a nice side effect.


Sure enough, launched SSMS 2012 and was able to login. And guess what, my login has sysadmin privileges.


And so I’m sure some of you have already yelled “SECURITY HOLE!!!!” by now – yeah, to a degree…but remember kids, if you’re a local Administrator on the box, you already own the box. Very little applications like SQL Server can do to protect themselves from a “rogue” Admin. Maybe a few adjustments to the security model for Windows’ SCM (Service Control Manager) are needed here, but I’ll let you decide on that.

UPDATE FOR SQL EXPRESS USERS: If you need to leverage this trick on a SQL Server Express install, you can use the “Winmgmt” service – Windows Management Instrumentation, aka WMI service.



Tale of an Encrypted SSIS Package in msdb and a Lost Password

Yesterday a Developer at work asked for a copy of an SSIS package in Production so he could work on it (please, dear Reader – withhold judgment on Source Control – I know!). I logged on to the SSIS instance, and when I went to export the package…


Oops. I didn’t have that password. The DBA who uploaded the package to Production is long gone; my fellow DBA had no idea either – and the Devs returned a cricket sound when queried.

So I posed the obligatory question on #SQLHelp and a bunch of folks jumped in – some to help and some to make fun of me (thanks, @SQLSoldier @crummel4 @maryarcia and @sqljoe). I tried their suggestions to no avail…even ran some queries to see if I could figure out how to extract the package XML from the system tables in msdb:

[sql]SELECT CAST(CAST(p.packagedata AS varbinary(max)) AS varchar(max))[/sql] [sql] FROM msdb.dbo.sysssispackages p[/sql] [sql] WHERE p.name = ‘LePackage'[/sql]


This just returned a bunch of XML with encrypted data on it:


I knew there was a job in SQL Agent scheduled to execute the package, and when I tried to look at details on the job step I got the following:


Not very helpful.

The password had to be saved somewhere, but where??

All of a sudden I remembered that there was a system table I hadn’t queried yet:

[sql]SELECT sjs.command[/sql] [sql] FROM msdb.dbo.sysjobs sj[/sql] [sql] JOIN msdb.dbo.sysjobsteps sjs ON sj.job_id = sjs.job_id[/sql] [sql] WHERE sj.name = ‘Run LePackage'[/sql]

The result:


“Well, that’s really secure”, I thought to myself.



A (Late) Meme Monday Post: On SQLFamily


Yesterday a member of the SQL community who I deeply admire sent me a DM on Twitter asking whether I had done a SQLFamily post for Thomas LaRock’s (blog|@SQLRockstar) Meme Monday for November. I replied that I did not, and I regretted not having done so. A subtle DM followed my response: “Get on it, you have all week”. And indeed I must. So here’s an attempt to express some of my feelings on a community that has catapulted my career like nothing else before I embraced it.

Nanos Gigantium Humeris Insidentes


I stand on the shoulders of giants. My SQLFamily has given me support at all levels. Professionally and personally. There is never a lack of will to help and provide advice to others in this community. And I do my best to help. On #SQLHelp on Twitter, via email, or even on the phone. I expect no retribution, because I know that when and if I do run into problems, my SQLFamily will be there for me.

I have met some of the most humble, dedicated and most professional people in the SQL community. And some of them have pretty big titles: MVPs, MCMs, Regional Mentors, and even leaders of PASS, SQLCAT members, and even PMs and Devs on the SQL Server team.

All are welcome, and that includes YOU!

I have also met some people that are rather reserved and don’t participate as much in the community, for whatever reason. Be as it may, let it be know to all that we are a very welcoming community – heck, some of my closest friends and people I can count on in the community have completely opposite political views. We share one goal: to get better and help others get better.

Even if you are a lurker – my hope is that one day you’ll decide to give back some of what you have learned.

You have to take it to the next level

On one of my previous jobs as an IT Supervisor I used to tell my team all the time about the benefits of continuous education and self-driven learning. Shortly after I left that job, the company went bankrupt and some of my staff got laid off – some without any severance pay whatsoever. I eventually found out that some of them had a really hard time finding another job, because their skills were simply outdated. They had become stale professionals. Don’t be one of them.

If you don’t take advantage of these learning resources, somebody else will – and that person has an advantage over you when applying for that awesome job position that got opened. There’s a severe shortage of good DBAs and DB Devs out there. What’s your excuse for not being excellent?

Even if your knowledge of SQL Server is at the beginner level, really – you have no excuse to get better. Just go to SQLUniversity and learn from there. Don’t get stale!

Thank You

To all of you in the SQL community who put so much time and energy into helping others, my deepest gratitude to you. I can’t wait to meet you all again at the next event and share our SQL stories over a pint of beer (or a shot of Jaeger)