It’s no lie; I’ve been chosen to speak at the Professional Association for SQL Server Summit later this year in October. Sure, the announcements came out a few weeks ago, but I wanted to wait a little while before blogging about it. First, to process the whirlwind of emotions I was having about it. Second, to make sure that I wasn’t going to get a follow-up email stating that the Program Committee had made a mistake…

I’ve been chosen to give a standard 75-minute presentation in the new Cloud Application Development and Deployment track. Here’s the abstract for the session:

Automating in the Cloud

Looking for a way to test your backups? Need to find an offsite location as part of a disaster recovery plan? What about defining your backup retention scheme? You can handle these common DBA challenges by using cloud services in combination with PowerShell automation.

In this session, we’ll review the basic cloud architecture and then dive into the specific components of the solution to see the power in combining different kinds of automation. No prior knowledge of PowerShell or cloud services is required.

A DBA’s Reactions

Yes, that title was intentional.

As I mentioned, I processed quite a number of emotions when I got the email. The first was frustration: I was out to dinner when it came in, and I knew that it was in my inbox, but my phone’s connection was so bad that I couldn’t load the message at all. Literally, I wandered around the parking lot, holding my arm aloft on the quest for moar bars.

Once I loaded the email, unbelieving joy. Speaking at the Summit has been a personal and professional goal ever since I had the chance to attend back in 2010. I moderated Lightning Talks that year and had a hell of a good time doing it, but it’s different from presenting a session where people will attend to gain knowledge. And sure, I joke about waiting for that follow-up “mistake” message, but that’s just the self-depreciating part of me.

Finally, once I started hearing about results from others, guilt. There are many professionals… friends… who did not get accepted to speak. Friends with more experience, friends who are MVPs, friends who have spoken at the Summit previously. It’s not something I try to think about too much, but I do accept that the guilt is there.

I’m not going to turn the rest of this blog into a consolation or “here’s how to do better next time” post. All I’ll say is that I hope that you still find a way to the Summit, because I don’t get to see all of you very often, and I’d dearly love to hang out again.

Finally, if you’re running a SQL Saturday or User Group within four hours of DC and are interested in a Cloud-PowerShell mash-up, we should chat – you’ll get a bit of a Summit preview and I’ll get some preparation, because as they say: Summit is coming.

Sanametrix is hiring! You can check out the full listing on our website, but here are some highlights to whet your interest:

• Experience developing databases and ETL processes
• Great communication skills
• Interest in AWS, Agile, git, and statistics
• Solid understanding of SQL Server
• Base level DBA skills and tuning process

If hired, you’ll be working directly with me (that’s either a pro or a con…) within a larger team of .Net developers, QA, and a tech writer. We’re building an application (from scratch) for the National Center for Education Statistics (NCES) that will help them prepare data submitted by reporting agencies within each state for publication and public consumption. And we’re doing this all in AWS’ cloud.

As for Sanametrix as a company, I can tell you that they’re solid. They’ve supported and shown interest in my professional development and my work with the SQL Server community, which is more than I can say for other employers I’ve had in the past. And when I say “they” I mean the president and COO of the company. They are regularly available and always willing to listen to new ideas and improvements.

So if any of this has perked your interest, feel free to submit a resume to Denise. Or if you’ve got any questions, just send me an email and I’ll do my best to answer them for you. I’m looking forward to hearing from you soon!

After a little bit of thought, I came up with this work around. Granted, this isn’t a “word for word” how it works in the package at my job, but it’s certainly a close representation of what’s going on.

Typical SSIS Lookup

Typical Control Flow in SSIS

In this typical loading package, we’ve got a For Loop that will run the Data Flow task for each of our years.

Typical Incremental Loading Data Flow

Looking into the Data Flow, here’s a basic set up. We can use our parameter in our query for our Source, but regardless, our Lookup will read and cache the whole table before passing on our No Match rows to the Destination.

Typical Lookup cached the whole etl.SalesOrderHeader table

Since we’re iterating over a number of years, that Lookup will read and cache that table for each loop. If you are dealing with a large table (and large is relative to data set and server resources), then you might be in for a good, long wait.

Faster SSIS Lookup

In this improved version, I decided that it might be faster if I could use an extra table in order to pre-store just the rows I’d need for each iteration. You’ll see in the Control Flow, I’ve added something new to the For Loop container.

Modified Control Flow to add and load a staging table

I’ve included an Execute SQL Task, which will truncate my staging table, and an additional Data Flow Task, which will fill it with just the records that I need based on the parameter input.

Point the Lookup at the staging table instead

As far as the other Data Flow, the one that actually is checking and moving the records we’re interested in, the only change I’ve made is to the Lookup. Instead of querying the destination table, it queries the staging table, which is holding the subset of records we’re looking up in that iteration.

The lookup cached the exact number of rows, since that’s all that was loaded in staging

While results may vary, for my particular package and purpose, I was able to reduce the run time of my SSIS package from over an hour to just ten minutes. That’s improved the run time by roughly 83%.

Last week we published information on PASSDC’s upcoming meeting on June 13, 2013. We’ll be featuring Jorge Segarra of Pragmatic Works. Here’s the full presentation run-down.

Do More With Less: Consolidate and Virtualize

In today’s economic times, we are being asked to do more with less. Less budget, less hardware and yet our data needs keep growing! So how do we keep our database infrastructure humming along given these restrictions? In this session we will talk about different options of consolidation including: database-level consolidation, instance-level consolidation, and virtualization. Additionally we will also talk about some of the tools and methods you can use today to help you consolidate your SQL Server environment and do more with less!

In addition, we’ll be holding an after-meeting event called Office Hours. You’re not ready for the fun to stop and neither are we, so join us across the street at the Capital Grille for a drink while we discuss all manner of SQL Server topics.

We hope you’ll be able to join us this month as we continue to grow the DC area community. Don’t forget to RSVP so we know you’re coming. See you on June 13th!

Automating in the Cloud

Looking for a way to test your backups? Need to find an offsite location as a part of a DR plan? What about defining your backup retention scheme? You can handle these common DBA problems by utilizing Cloud services in combination with PowerShell automation. In this session, we’ll review the basic Cloud architecture, as well as some specifics as to the components of the solution.

The nice part of this session is that I’ll be talking about both Amazon Web Services (AWS) and Windows Azure. The focus isn’t really to point out any differences (though they exist), but it’s to show that regardless of your chosen Cloud provider, you can create an automated solution that can work for your company.

Again this is late in announcing (cause I’ve been working hard on the session!), and I’m still pulling together the presentation resources page, but I do hope we’ll be able to meet up this weekend in Philly. If not, perhaps I’ll see you next weekend in Columbus, OH!

Part of the Gulf Coast

I wanted to share a quick Friday update to mention that I’ll be presenting Build your Own Virtual Lab at the Gulf Coast SQL Server User Group on May 23, 2013. As much as I’d love to visit in person, I’ll be joining my peers over webinar that evening. I think that we may end up using Google Hangouts, and if we do, I may be able to record the session directly to my YouTube channel. So I’ll be sure to keep everyone updated on the status of that as we parse it out in the coming week.

I’ll see you wonderful Gulf Coast folk next Thursday!

T-SQL Tuesday

It’s time for T-SQL Tuesday the forty-second, and Wendy Pastrick has asked that we talk about our experiences with change in our working lives, specifically as it relates to technologies we’re using. I’d like to share a bit of my own history and relate it to managing change.

Career Milhouse

Post graduation (and even pre-graduation), I was landing typical gigs as a temporary employee: admin jobs, stuffing envelopes, and data entry. Jobs that paid bills and allowed me to eat. While it provided me with a good amount of experience, the jobs were unfulfilling. To make the issue worse, I lied to myself about the situation: "As long as I have personal time for my own projects," I’d say, "then I’ll be happy with whatever pays the bills."

I was attempting to cover up the fact that I was holding a series of dead end jobs. Positions that were unchallenging, and didn’t have any means of growth or promotion. As a consequence, my work life tended to be lifeless.

I knew I had wanted to work in technology for years at that point, but didn’t know how to manage or act upon it. And it didn’t help that most "entry level" tech jobs required "two or more years of experience." In that place in my life, I had a mismatch between my goals and my ability to meet them, which isn’t to say that I’m special in this regards. I think many twenty-somethings have the same experience as they stumble about post-graduation and figure out how the world works and their place within it.

Simply, I was in a dead end, had a goal in mind, but no way to prioritize it to move forward.

Drowning in Technology

After the happy accident that turned me into a DBA and technologist, I still ran into challenges. Burnout on the job, learning to manage "poser" feelings in relation to the greater community, and balancing both a career and family became the new obstacles that shifted my priorities. But that’s common: everyone deals with that.

Technology itself often becomes a burden for me. I have a love of learning, and a love to figure out how technology works, but these traits aren’t solely positive. There are repercussions to being a "jack of all trades." The main problem is that you’ll spend time away from your primary focus, which in turn will delay your professional growth. And with the exception of DevOps, technical generalists aren’t ever in demand.

On a side project, it had been decided that we would build an app using Python on top of Django. "Perfect!" I thought. "This will be a great chance to build some basic programming skills and learn more about MVC!" And it was. But in the scope of a data professionals career, how helpful are those base level skills? While I spent only a few months on that particular project, I’ve not used Python or MVC since. What SQL Server learning opportunities did I miss in the meanwhile?

On a more hostile note, when I was a contracting I had the misfortune to troubleshoot deep issues with SharePoint. I was the only person on staff capable of the work; project managers were spinning and playing the blame game. While we all failed from a customer service perspective, I was still successful in closing out the issue. But since the work was so specialized, the knowledge I gained from it wasn’t useful towards my larger goal of becoming a more competent data professional.

As in all things, there is a need for balance. On one hand, we need to maintain a base-level knowledge of new and upcoming technologies to remain competitive in our careers. But always chasing the shiny new hotness can lead to dead ends slowing down your growth instead.

Finding the Right Mix

In all honesty, it wasn’t until the end of 2012 that I figured out how I wanted to focus my career, and started to lay out concrete goals for myself. And even before I started laying those goals, I had to come to a difficult decision about where my true technological passions lied. And in the end, it was with data. It had always been data. (I apologize for this having turned into a sappy love note.)

While I cannot deny my love for learning, I’m maintaining a greater focus on technologies that are closer to my chosen profession, and with an eye toward the future. That’s why I chose to take my current job with Sanametrix, because I could pair my career in data atop of AWS‘s cloud platform. While I didn’t work directly with Obama’s AWS team, I saw the results of their work, and it changed how I thought about availability and serving end users. It is the future, and the future is here. Returning to my theme of balance, it’s not that the cloud will ever fully replace the enterprise (at least foreseeably), but that they’ll be used in conjunction to meet business needs.

It’s in this latest job that I’m finding an alignment of my priorities and my goals. And thanks to the continued growth of data and the cloud, I’m staying well away of dead ends.

Communicate for Great Good!

“No one knows what I do.” That’s a chief complaint of data professionals everywhere, and it’s a dangerous position to be in as well. Reviews may not be favorable, promotions more uncommon, and training dollars more rare. But by practicing written and verbal communication, you may be prepared to take advantage of new opportunities that present themselves. This presentation is for data professionals looking for fun activities to grow their written and verbal communication skills.

I’d love to have you attend. Typically the session is laid back and interactive, and I’ll do my best to keep it that way over the webinar. And if you haven’t already, do hit the link for the Professional Development chapter as they offer great content every month, and it’s one of the easiest venues to find new ways to further your professional growth.

See you next Wednesday!

The Setup

My struggle came about as I was setting up an off-site backup location on AWS’ Simple Storage Service (S3). This is something I’ve blogged about in a past T-SQL Tuesday, and I’ll be talking about in an upcoming presentation. The trouble was that my PowerShell script would no longer move the .bak files to S3. The even more troubling part was that I hadn’t made any changes to the server.

If you haven’t had the chance to work with PowerShell in the past, I can only describe the troubleshooting process as obtuse. Perhaps with a great deal more skill it becomes simple, but as a novice I wasn’t sure where to begin, especially because my SQL Agent jobs were returning successfully. My technique involved printing my variables and parameters to file until I narrowed it down. Thankfully this worked, and I figured out that, while I had stored my AWS credentials properly, they couldn’t be accessed by the SQL Agent job.

The Details

According to the engineers at Amazon, the API credentials are stored on a per-user basis. This means that while I could RPD into the server and do my normal setup, those credentials would then be tied to my login. This is sensical, but makes setting up administration a bit more tedious as you have to set up the same credentials for any other user accounts that may need them. In my case, I needed to perform additional setup for my SQL Server Agent account, since I’d be calling my PowerShell script through the Agent.

The interesting part is that Amazon enforces this per-user basis in how they store the credentials. Within each user’s appdata\Local folder, there is a .json file that has the credentials. But the credentials are (appear to be) salted and hashed to tie them to each user uniquely. So while copy-pasting the .json file from one user to another allows the new user to see the credentials through a call to Get-AWSCredentials -ListStoredCredentials, they won’t be able to use said credentials.

The Fix

So how do you fix this so the SQL Agent can store the appropriate credentials? Well, I just wrote a new job to store the credentials, similar to what I did in my original post, and then I deleted the script and the job afterwards for security’s sake. Really, you wouldn’t want to leave the API credentials in a script or even a PowerShell $profile. And you can bet that I’m going to be looking for a nice “PowerShell” way to script the credential setup on new machines. I’ll share once I crack that nucket.

Wanted to send out a short blog sharing the information for PASSDC’s upcoming meeting on May 9, 2013. We’ll be featuring Ted Malone of Microsoft speaking on data analytics. Here’s the full presentation run-down.

Business Analytics in a Big Data World

“Big Data” is a topic that everyone is talking about, but few actually have a clear definition of the Big Data problem space, or how Big Data Analytics fits into an overall Business Analytics process.

In this session, we will discuss how Business Analytics and traditional Business Intelligence systems fit into the Big Data space. We will also discuss the Microsoft approach to solving “Big Data” problems. We will demonstrate capabilities of the Microsoft Federal team’s “Enterprise Data Analytics Platform” and show how familiar tools can be used to gain insight from data, no matter the type or location.

In addition, we’re starting a new after-event called Office Hours. You’re not ready for the fun to stop, and neither are we, so join us across the street at the Capital Grille for a drink while we discuss different SQL Server topics.

We hope you’ll be able to join us this month as we continue to grow the community of professionals in the DC area. Don’t forget to RSVP so we know you’re coming. See you on May 9th!