8/10/2016

Refreshing SQL Server development databases

Refreshing the Dev environment

I started a new contract recently.  This is a global company, but I am on a relatively small product team within it.  10 developers, 3 admins (SQL, O/S, Storage), 2 Business Analysts.

The company has a formal Change Management process, which is great.  However, this team manages and operates outside of that structure...not so good.   Devs have sysadmin to production. For now.

I and the other DBAs are working on all of the things you would expect to be missing from this sort of environment...consistent backup strategy, documentation, proper security, etc.

As with most places our Developers have a Dev copy of the database and are told not to touch the Pre-Prod version.  But, since we are test loading data there, Dev is way out of date.  Also, there are objects in Dev not moved to Pre-Prod yet.  So, they regularly point their queries, procedures and even the website to the PP database.  And then forget what they are connected to after lunch.

This makes for interesting hallway conversations!

The solution of course is to refresh the Dev db from Pre-prod, without losing any Dev code or users.

DBAs that have been around for awhile know where I am going with this...

Basic steps:

1. Backup the Production (or Pre-prod in my case) database.   Or pull from your overnight backup.   You DO have an overnight backup, right?   RIGHT?!?!?   ;)

2. Restore that backup to the Dev server as MyDB_Refresh:

-- Get the logical and physical filenames
Restore filelistonly
From disk ='B:\Backup\FULL\MyDB_FULL_20160810_000012.bak'

---Restore, not overwriting the existing Dev db
Restore Database MyDB_Refresh
From disk ='B:\Backup\FULL\MyDB_FULL_20160810_000012.bak'
With move 'MyDB' to 'B:\Data\MyDB_Refresh.mdf'
       ,move 'MyBD_log' to 'B:\Log\MyDB_Refresh_log.ldf'
       ,recovery

3.  Use SQL Server Data Tools/Visual Studio, or a 3rd party tool such as Red Gate SQL Compare to compare and synchronize the schema in MyDB (Dev copy) to MyDB_Refresh.  SSDT works well for this if you don't have a license for a vendor tool, or don't want to download a trial version.  The Refresh database is the Target...

4. Once you have the tables, procedures, USERS, and other objects exactly the same as the old Dev database, you need to rename both databases:

--Make sure all connections are closed:
Use master;
Go

exec Sp_renameDB 'MyDB', 'MyDB_Old'

exec sp_renameDB 'MyDB_Refresh', 'MyDB'

If you are having trouble getting exclusive use because Devs or apps keep re-connecting, you need to use a 2-step process (run both at the same time):

--Take the database offline and kill connections
Alter database MyDB Set offline
with rollback immediate

--Bring it back online and rename all at once
Alter database MyDB set online
use master;
go
Sp_renamedb 'MyDB', 'MyDB_Old'


Change the database names if you need to do the same on MyDB_Refresh.  You can use the same offline/online code when you are doing an emergency restore in prod and the app servers won't let go of their connections.

Compare a select of tables between Prod and Dev to make sure the data is refreshed, and double check using schema compare that Dev is different than Prod (likely, since that is what Devs do...).

This is a basic process, best done in a maintenance window, early in the morning or late at night.  And you can automate most of this.  I have not tried to automate schema compares other than using SQLPackage.exe in the SSDTools to "upgrade" a target data-tier application, or create a Drift report.   But that is way beyond the scope of this post.  Maybe another time :)

Please feel free to post up your processes, or enhancements to this one that some new DBA or Windows admin may be able to use 3 years from now when he finds this post on Google.

Have a fantastic day!

Kevin3NF


8/09/2016

Join vs. IN Not IN

This post has been moved to:

http://dallasdbas.com/join-vs-in-not-in/

Come on over and look around!

8/04/2016

Login Failed for user...

This post has been moved to my site:

http://dallasdbas.com/login-failed-for-user/

Thanks for visiting!

6/01/2016

SQL Server 2016 Stretch Database

New to SQL 2016 is the ability to send archived data off-premises to MS Azure storage, in the form of a "Stretch Database."  Sounds like a great idea in theory but do your testing and bust out the calculators before you put production data in the cloud. 
Things I like:
  • Not buying storage, especially on a maxed out server
  • Easy to query full dataset across on-prem and Azure stretch
  • Ummm...all editions is a good thing instead of Enterprise only
  • Nope.  That's it.
Not fond of:
I'm pretty sure I could roll my own "stretch" function into a Azure SQL Database, and I'm an admin much more than a developer.
Maybe down the road this will be better, but right now its an expensive alternative to a USB drive from Fry's, or a NAS/SAN upgrade.
Are/were you planning to use Stretch?  Have a differing opinion?  Let's hear it!
Kevin3NF

2/19/2016

Dudes...really?


I was participating in an email interview about my SQL career and opinions and one of the questions was basically "What's the worst SQL thing you've inherited?"

I gave them this list:

o   SQL 2008 R2, RTM
o   Incorrect Memory configuration
o   Full recovery model on data that only changes once a week at most
o   ZERO documentation
o   New data is imported to a new table and a UNION ALL is modified to add that table
o   ZERO documentation
o   Stored Procedures have no comments, poor formatting and developer names in the name of the sproc
o   Autogrow is 1MB, data imports are hundreds of MB each
o   Everyone is sysadmin, probably including you…
o   Change control process is intentionally shortcut and ignored on this internal production system

o   Ownership changed to me in December, then was yanked back 3 weeks later with developers overwriting my fixes in prod.

Really....all that on one server!

Upgrade your fries broseph!!!

Kevin3NF

2/11/2015

Unsupportable...thats what you are....

With apologies to Nat King Cole :)

Database backup failing

Write on "D:\backup\MyDatabase\MyDatabase_20150208104103883_D.bak" failed: 665(The requested operation could not be completed due to a file system limitation) [SQLSTATE 42000] (Error 3202) 

Hey, customer...let not put backups on O/S compressed drives please.

Oh...and why do you have your MDF/LDFs on the same drive...also compressed?

Resolution - punt it back to them and tell them SQL Server on compressed drives is unsupported config by Microsoft and by me.   Uncompress and expand or add a drive and uncompress.

Sheesh.  How many worst practices can you get in one server?

5/03/2013

Monitor for blockings...

Short and sweet...

Ticket for error 1105.   Teammate response: No blockings on server, monitor 4 days.

Ticket for error 601.  Teammate response: No blockings on server, monitor 4 days.

Ticket for login failed.  Teammate response: No blockings on server, monitor 4 days.

I.  Kid.  You.  Not.

Someone needs to be fired.  Or congratulated for making me look like a SQL SERVER GENIUS!!!! by comparison.

That is all.

Kevin3NF

4/15/2009

OK...if all 3 instances on a 3-node cluster suddenly start "timing out", yet you can log directly in and query the databases...don't you think you should at least look at the network components first?

Especially when the errors are showing on the client and web boxen...

Grrrrr....

3/11/2009

When to Modify Somebody Else’s Code

I was tagged by Brent Ozar on this question that has been floating aroung the Tweeterverse and blogosphere the last few days...

In keeping with the short-but-useful-or-amusing style of this blog:

When you have an OK in writing from the vendor or internal group that wrote it AND from the supervisor you got the approval from on your side.

Yeah...make darn sure your suggestion actually works before you even ask, do backups, keep scripts, etc.

Else you get no fries. You get fried. :)

Kevin3NF

Tag: Lee Everest

1/29/2009

You did what?

This one comes from inside....

We got an alert that the Log Shipping was out of sync for a customer. Our front-line guy looked and saw that the LS Restore job was still running after 35 minutes when it normally takes less than 30 seconds.

He decided it was hung and cancelled it. Restarting the job caused immediate errors.

Why? The restore was working on a 1.4 gb t-log backup file, instead of the normal 4-5mb. Cancelling it puts the database in a suspect state. had to drop the db and re-initialize from a full database backup.

Ugh. Glad it wasn't any bigger than 7gb...since the file/restore had to move halfway across the country.

No fries for you "M from M"...you owe me some!

Disclaimer: this is a pretty sharp guy and I'm just harassing him here...

Kevin3NF

1/15/2009

North Texas SQL Server User group

I live, work and play in the DFW area and used to be a regular attendee at the local user group monthly meeting. I was even a board member. The topics became increasingly over my head or out of my area of daily need, so I started slacking and eventually quit going altogether.

No more!

Tonight I start the first of 5 or 6 months of free Power Shell beginner classes that happen before the SSUG meeting starts, taught by a local expert. I don't even know what Power Shell is...some sort of scripting language is all I gather.

In addition, my co-worker Trevor Barkhouse is the presenter this evening. He will cover "Refactoring T-SQL Code for Better Performance". I've seen him do this here in the office to help a couple of our problem customers resolve nagging issues that were frustrating them.

More about NTSSUG

Kevin3NF

12/16/2008

A series of conversations from several days...compacted into one nightmare. Enjoy:



Front line dude: Acme has a drive space issue...can I shrink the log file?

Kevin3NF: Give it a shot (this is SOP for Acme on this drive)

FLD: Didn't work

Kevin3NF: Give me the ticket, I'll take a look.

Kevin3NF: Hey ACME developer...you have a 2 day old transaction taking up all your T-log space...

Acme: ok...let me truncate the log

Kevin3NF: WAIT A SECOND!

Acme: Did that help?

Kevin3NF: No. You just invalidated the T-log backup stream and started causing the backup to fail

Acme: Why?

Kevin3NF: Cuz that big open transaction is still there, and SQL Server thinks there is no full backup now.

Kevin3NF: Hello?

Kevin3NF: (Sees pictures of crickets in his Inbox)

Acme: (hours later): go ahead and run a full backup tomorrow afternoon

Kevin3NF: You know you don't have a valid backup to recover to, right?

Acme: Yes, but we can't backup now...

(tomorrow afternoon):
Acme: Hey...who told you to run a full backup?!?!?

Kevin3NF: You did...I have the email.

Acme: (complaint to Kevin3NF's bosses)

Time passes...repeat scenario.

Big transaction, full log file, fill drive, lather, rinse, repeat.

Customer: FAIL. >:(

Acme me owes ME some fries...but I get paid to deal with stuff like this every day :)

Kevin3NF

11/25/2008

SQL PASS Community Summit 2008

Wow. What a load of information! Almost an overload, but managed to keep my head above the water in almost every session.

The breakdown:

Day 1:

SQLCAT - Overall Lessons Learned from SQL 2008 Experiences
Quite a good bit of variety here, touching on:

   Data Compression

Backup Compression

Performance Tuning (Extended Events)

SQL Server Reporting Services

Database Mirroring (compressed LogStream)

Encryption

Capitalizing on the SQL Server 2005 System Information
Way more info here than I was able to process...took a few notes to look up after Thanksgiving


Understanding and Troubleshooting Transactional Replication Performance
Lots of discussion on subscription streams/mutli-threading and read/write activity of the Log Reader and Distribution Agents. Got a few things to play with.


SQL Server 2008 Policy-Based Management - Sharon Dooley
Not a whole lot here I didn't pick up at the SQL 2008 launch in LA...still a very cool feature I won't get to use much in our environment.


Day 2:

Guiding your Query Plans (by Kalen Delaney)
Guiding query plans didn't actually enter the picture until late in the presentation, but it was fantastic nonetheless. Most of this session was about Hints...Table, Index, Option, etc. A BUNCH of stuff that never crossed my monitor before. 3 pages of notes worth. Being able to force a recompile of a portion of the overall query is really slick. And the very simple new feature of assigning a value in the declare:


DECLARE @MyID INT = 123


Smart Database Design - MVPs DrSQL and Paul Nielson
Well outside of what I do from day to day, but pretty cool anyway, since I was a developer many moons ago. Worth the lack of elbow room just to hear from DrSQL and Paul live...too bad my on call phone went a little nuts and I had to leave early


Advanced Troubleshooting with SQL Server Extended Events
So far over my head I'm still not sure what I learned...good thing I have notes


Business Continuity with Backup and Restore - Peter Ward
I could probably have taught this session. Should have been labeled 200 level, but still good solid information on backup/restore and some of the new (05/08) options (Online restore, Partial, COPY_ONLY, etc.). Thought about arguing a point with Peter, but decided the way he presented his wasn't likely to cause harm.


Day 3:

Data and Backup Compression Lessons Learned

Some fairly impressive numbers here...300GB DB compressed to 45GB, 50% time savings, etc.

Money-maker: use sp_estimate_data_compression to do custom analysis for customers considering whether to move to SQL compression. Assuming you have the install to do it on.

Collecting and Analyzing Performance Metrics in SS2005/8
Got here late, stuck in the back, couldn't see or hear very well. Saw a little perfmon on the screen I think...


Writing Technical Articles (Kathi Kellenberger)

The only Personal Development session I attended, since I was on the company nickel. Interesting stuff. I've thought about teaching and writing a SQL Basics series for new or accidental DBAs that really will never do more than backup/restore or set up Dev systems. Kathi Kellenberger did a fine presentation, including various places to publish to, and what they pay (if anything). I'm completely convinced to NEVER get in on a book project :)


End-to-end troubleshooting for SQL Server 2005
Kevin Kline rocks. Many others have blogged this presentation, so I won't repeat here.




11/17/2008

PASS Summit 2008

Heading out tomorrow morning....feel free to grab my arm and say hello if you see me there. I'll be the 40 year old slightly overweight dude sporting a laptop in a backpack ;)

Kevin3NF

11/11/2008

I hate it when that happens...

A good friend of mine has a fairly simple community site, with 30K members, about 7K active.

SQL Server 2000, ASP Classic. DB and site on separate servers.

KerBam! SQL Injected last Monday. Major trashing of data.

Problem 2: No backup since October 2, as the SQL Agent password had changed so the Agent wasn't started.

Problem 3: Started the agent and it deleted the last full backup because it was older than 4 weeks.

SO: I get the call for help.

Time passes.

More time passes.


After a dozen uploads and downloads of .mdf/.ldf and backup files (from August), I am able to recover much of the data using the fantastic tools from Red-Gate software:

Log Rescue: Identified what got injected, into where and when
SQL Compare: enabled me to create a schema script to replace the relationships I had to remove to fix the data
SQL Data Compare: Helped me replace the trash data with what it looked like in August.

Also, a shout-out to Narayana Vyas Kondreddi for his Search and Replace code that at least got the bad URL out of the data we couldn't fix.


The site is back up in read-only, and my buddy is reviewing all the code one page at a time.

Lessons learned:
Validate your inputs!
Back up your data, and verify it!

Not only does this one get fries, he gets 10+ hours of recovery effort at no charge, just for having a really cool site I want to see come back up :)

Kevin3NF

9/17/2008

Do what?

Too good to be true, but it is...


From: My Customer
Sent: Tuesday, September 16, 2008 9:37 AM
To: Kevin3NF
Cc: a bunch of people
Subject: Re: SQL Backup

Hi,

As per latest discussion with {Customer DBA} only one database will be backed up - "MyDatabase". Please exclude other three databases from backup process.

Differential backup should be from Monday [after taking of full backup] - Sunday.

-Customer

----- Original Message -----
From: Kevin3NF
To: My Customer
Sent: Tuesday, September 16, 2008 9:41 AM
Subject: RE: SQL Backup

Full backup at 12:15am and differential 15 minutes later at 12:30? We can do that…

----- Original Message -----
From: MyCustomer
Sent: Tuesday, September 16, 2008 9:43 AM
To: Kevin3NF
Subject: Re: SQL Backup

yes that's correct.





Apparently they anticipate a lot of traffic the first 15 minutes of Monday morning each week :)

Sigh.

Kevin3NF

8/07/2008

So SQL 2008 went RTM yesterday...cool.

Now, to dig in and see what (if any) of the really cool new features moved to the standard version instead of Enterprise. Or even stuff from 2005 moving into standard...

Post a comment if you already have the list :)

OK...here we go:
Features Supported by the Editions of SQL Server 2008
http://msdn.microsoft.com/en-us/library/cc645993(SQL.100).aspx

Backup compression: Enterprise only. No thanks, I'd rather buy Litespeed or Red Gaet SQL Backup than pay this premium :(

Encryption: Enterprise only (TDE and Data Compression): Great feature, easier to use, worth the expense for some companies/industries

Resource Governor: EE only...cool feature, but I wonder how often it will get used in practice

Database Mirroring: Yes (safety full only) in Standard edition

Auditing: EE only. Someone is going to blow performance out the door by auditing everything...be judicious here folks!

Capacity Specifications for SQL 2008 objects:
http://msdn.microsoft.com/en-us/library/ms143432(SQL.100).aspx

Maximum Number of Processors Supported by the Editions of SQL Server 2008
http://msdn.microsoft.com/en-us/library/ms143760(SQL.100).aspx

A Processor is still defined as a socket here, so 4 quad cores gives you a SE SQL Server with 16 CPUs.

Memory Supported by the Editions of SQL Server 2008
http://msdn.microsoft.com/en-us/library/ms143685(SQL.100).aspx
Workgroup edition got a 1gb bump here

And my favorite feature thus far is still the very configurable installer. I install at least 1 SQL Server per week and despise having to physically move the system databases and set the default locations after he install for the user dbs. There are (too?) many options in this installer that were simply not available pre-2008. Rockin'

So, there's some links and my take on a few pieces.

Fries and a shake to MS for not slipping the date to Q4!

Kevin3NF