Category / Disaster Recovery

by Samuel Kastberg

Automating BizTalk Server installation and configuration

Installing and configuring BizTalk Server can be complex, time consuming and error prone. The complexity comes not from the process itself but more from all the different components and possible configurations. My objective is to share my experiences from working with several customers and some techniques you can use to create your perfect installation. The objective is not to show “the perfect” process, perfect for me might not be perfect for you. In this post I share an overview, later I will do some more with more details and share some of the functions I use.

Before you start

Decide what your main drivers are and let that be your guide through the creation. Repetition and control are generally the drivers to automate, and the goal standardized developer machines, disaster recovery preparation, test environments or ensuring the environments are equally configured over our Dev to Production pipeline.

Decide what your baseline is and document it, think what could change in 6 month or a year. With one customer we created a brilliant setup that started with bare Windows installations and within 2 hours a highly available solution was in place. Discussing a year later we concluded that creating the LUNs and volumes might have being overdoing it since the underlying storage will be changed. It would have been better to have it as a requirement in our baseline or as a distinct step in the process that could be very easily changed or replaced.

Consider internal organization and politics, i.e. if you will never be allowed to install SQL Server or create groups put it in your baseline document as a pre requisite.

Set a timeframe for your work, if not you can spend weeks searching for perfection instead of reaching your goal. If you’re too ambitious you might end up in a overly flexible process that just is a parallel to the normal one, not a good idea.

Document the execution process. Write the running order of scripts and shortly what each one does.

Windows

Generally, Windows installation is already taken care of and I see it as a part of the baseline. That said, you should ensure that the features and configurations you need are in place. You will need the Windows installation source and can use PowerShell Cmdlets like Enable-WindowsOptionalFeature or Install-WindowsFeature (will not work on Windows 10). I find this post good to find features and decide which one to use.

Sample Script to enable Windows Feature

BizTalk Server

Setting up BizTalk Server product consists of two parts, installation and configuration. Installation will add the binaries to the system. Configuration will create or join a BizTalk Group and enable/configure other features as Rules Engine.

When running Setup.exe with the /S command it will use the list of InstalledFeature elements in the configuration file you specify. The silent installation details are documented here.

When running Configuration.exe with the /S command it will use the Feature elements. Each Feature element represents a section in the configuration dialog box. I will look more in depth on this on another post.

Sections of the configuration file

Installation of additional software like Adapter Pack, CU/Feature Pack, WinSCP (needed for SFTP adapter) can also be installed silently. Setting up hosts and handlers can also be done.

SQL Server

SQL Server can be silently installed and use configuration files to use the configurations you will need. I leave this with a pointer to the documentation.

Things I have scripted post installation are

  • Setup of Availability Groups and creating empty BizTalk Databases with the file setup I want to have.
  • Setting in the Primary Check for availability Groups
  • Configuration of Backup and DTA Purge.

Wrapping up

Basically, all parts of setting up a BizTalk Server environment can be done with script. Your needs and environment set the limits. I believe scripting your environment is a good way to get to know the components you’re using. I will follow up with some more posts that will go more into hands on approach to the different parts.

I will do a session on this matter att Integrate 2019 in London https://www.biztalk360.com/integrate-2019/uk See you there!

by Samuel Kastberg

Generating restore scripts for BizTalk Server – Version 4

I have created a new version of my PowerShell script to generate restore scripts for BizTalk Server. The basic idea is the same and you can read more about it in my previous post here

I have rewritten the script and done some changes.

Download the script with some fixes for failed backups in the chain and secondary replica not readable RestoreScriptForBizTalkJob.4.1.0 

Support for Availability groups

With the release of BizTalk Server 2016 we can use SQL Server availability groups. The previous version of the script didn’t work with availability groups. I have added a function that checks if the running server is the primary replica or not in an availability group to return true and false if it is a secondary replica. The script just exits if it is a secondary replica.

Script parameters

The new script is intended to be called from the job step using parameters to the file.

Parameter Default value Comment
ServerName Name of the SQL Server including instance containing BizTalk Management database i.e “Server01\myinstance”.

Note: If you use the listener name in an AG you will need to add the instance name.

Mandatory
MgmtDb BizTalkMgmtDb Name of BizTalk Management database

Not mandatory
SqlScriptDestination NULL By default, the script stores the scripts in the full backup folder, use this parameter to redirect to another folder.

Not mandatory

 

Removed loading SQL Server PowerShell provider

The script does not load the SQL Server PowerShell provider, the reason is that I just wanted the script to be a bit cleaner.

Check if database exists

This version checks if the database exists before setting it to single user mode to avoid unnecessary error messages while recovering.

Tested scenarios

This version of the script has been tested with:

  • BizTalk 2016 using SQL Server 2016 with Availability groups
  • BizTalk 2016 using SQL Server 2016
  • BizTalk Server 2013 R2 using SQL Server 2014

I have tested both generation of the scripts and recovery.

Reminders

When you recover BizTalk server databases it is all or nothing. All the databases for the group that are included in the backup job must be restored together and to the same transaction mark.

While recovering, ensure that you have stopped all activity in your BizTalk group.

The script is licensed “as-is.” You bear the risk of using it. The contributors give no express warranties, guarantees or conditions. Test it in safe environment and change it to fit your requirements.

by Samuel Kastberg

Generate restore scripts for BizTalk Server backups

The BizTalk backup process

A BizTalk group’s data consists of a number of databases with data moving between them. To keep a consistent state when taking a backup, the job uses a marked transaction. A description of that process can be found here. Every time the backup job runs some rows are added to the adm_BackupHistory table in the BizTalkMgmtDb database containing information like the MarkName and files. This information is used by BizTalk Log Shipping to keep track of backups.

BizTalk Log Shipping

BizTalk Log Shipping is the only fully supported way of disaster recovery. When backups are taken SQL Server jobs will restore them to a passive server. You can find a description of BizTalk Log Shipping here and how to configure it here. There might be situations when you need to manually restore the databases. This process is both tedious and error prone, especially during a critical situation.

The task explained

You have a number of backup files of data and logs and need to restore manually. When restoring the last transaction logs, you must use the STOPATMARK argument to keep the databases in a consistent state. What can we do to be prepared in this situation? Having a script prepared to restore is a good way. The best moment to generate the script is directly after we have taken the backups. The idea is to add an extra step to the backup job that generates a restore script based on the information in the adm_BackupHistory table, using the same information as the BizTalk Log Shipping.

Note: You should never change the logic of the BizTalk Server jobs; in this case we’re augmenting the process without changing what the backup job does.

The script

How it works

What do the script need to do to create the restore script?

  • Get last mark name from adm_BackupHistory
  • Get last full backup file information from adm_BackupHistory
  • Loop all databases
    • Generate full database restore script for the current database
    • Get log file information for the current database from adm_BackupHistory
    • Generate log restore script for the current database using with no recovery for all except for the last one which have the current mark. For the file with the current mark generate script using STOPATMARK.
Configure the script

In the top of the script you have three variables that might need to be updated.

Variable Comment
$ServerName Database Server name where BizTalk management database is located. If it is located on a named instance the instance name must be provided.
$MgmtDb The database name for the BizTalk management database.
$SqlScriptDestination Default set to $null, then the generated scripts will be stored in the same location as the full backups. Set a path if you want the scripts saved on a different location.

clip_image002

Getting the script in place

1. Add an extra step to the backup job of PowerShell type. Add your version of the script in the command field or the full path to the PowerShell script.
Note: If you have implemented Log Shipping you must use a file since the script is longer than the 3200 characters that the Log Shipping process will copy.
clip_image004

2. Change the ”Clear Backup History” step to go to next step on success. Also set the new step to quit responding success or failure.
clip_image006

The resulting SQL scripts

For each backup a new set of script files is created, one file for each database instance that is included in the environment. The naming is “Server_Instance” (Green rectangle in the picture), “Markname of starting full backup” (Blue) and then the “last mark” (Red).

clip_image008

clip_image010
Each database will be set to single user mode and close all open connections. All the databases in the backup have a restore database using with move in the expression so you have the possibility to change the location of the database files. Every restore of logs have NORECOVERY except the last one that use STOPATMARK to stop at the transaction mark.

Restore process

When you are in the process of restoring and have an environment with no activity, I.e. all BizTalk Services and SQL Server agent turned off, you can run the scripts on the database instances where you want to restore. The scripts can be edited to have a different file location if needed. If you’re moving databases to another server don’t forget the normal procedure running the UpdateDatabase.vbs script with and updated SampleUpdateinfo.xml file.

The script is licensed “as-is.” You bear the risk of using it. The contributors give no express warranties, guarantees or conditions. Test it in safe environment and change it to fit your requirements.
Download the code and description 

Note:
This version only supports writing the full path to the script in the jobstep.
This version don’t work with Availability groups, a new post is coming soon.