Updating IP Addresses on VMware servers using Site Recovery Manager
The SRM process to update IP addresses on servers during a failover is straightforward and relatively well documented. SRM uses the VMWare Guest Tools to update the IP addresses, and can apparently work with any supported guest operating - Windows 2008 and CentOS both worked with no problems.
Updating DNS Records using Site Recovery Manager
The SRM process to update DNS records during a failover is poorly documented and has a few challenges.
VMware provides one kb article - if you don't fix their supplied file dns_update.cmd, you risk changing DNS records for your production server(s) during an SRM test failover, which could be a very big problem.
There are many more challenges to getting this process to work properly.
Running Site Recovery Manager as a Windows Service Account
If you want SRM to update Active Directory DNS servers, you will need to run the actual Site Recovery Manager service (Control Panel / Administrative Tools / Services) as a domain user with enough permissions to make DNS changes in AD. By default SRM is installed to run as the LocalSystem process. The right thing to do is create a domain user to use as a service account and login as to the SRM servers as this user when you install SRM, but if you're like me you didn't know that and now you've already got SRM installed using another account. You need to change SRM to now run as another user. There are two ways to do this:
1) Run a repair install of SRM.
2) Use srm-config - I was told that doing this wrong can cause serious problems but I don't remember the details. Running a repair install is safer.
Create an Active Directory domain level SRM service account.
Grant the new SRM service account permissions to change Active Directory's DNS.
Make that account a member of the local Windows administrators group on your SRM server.
Login to the SRM server with this new SRM service account.
If you have made any changes to the SRM scripts such as those in c:\Program files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts, archive them now before your changes get erased.
In Control Panel / Programs and Features, select VMware vCenter Site Recovery Manager, then select Change to run a repair install of SRM. The repair install updates the user that SRM uses to connect to SRM server on same site, and also updates certificates.
Once you have reinstalled SRM, you have also broken it. It won't run again until you fix the database access.
Assuming you are using MSSQL Server 2008:
Open SQL Server Management Studio
Expand the server
Right click Security, then select New -> Login...
On the General tab:
Enter DOMAIN\serviceaccountname for Login name
Select Default Database SRMdatabase
On the Server Roles tab:
Check sysadmin (public should already be checked; leave it checked)
On the User Mapping tab:
Check db_owner (public should already be checked; leave it checked)
Select OK to create the login.
Now you need to use installcreds.exe twice to change the username for the SRM database's ODBC DSN. You run installcreds once to remove the current database credentials, another time to set the new credentials. Installcreds only changes the registry, it doesn't change any database settings or the ODBC DNS
<snip>
C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\bin>installcreds -key "db:srm"
VMware internal use only. This program is intended for use only by the SRM installer.
Removed credentials for db:srm
C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\bin>installcreds -key "db:srm" -u DOMAIN\srmserviceaccount
VMware internal use only. This program is intended for use only by the SRM installer.
Enter Password:
Installed new credentials for db:srm
C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\bin>
<endsnip>
In Control Panel / Administrative Tools / Services, change the VMware vCenter Site Recovery Manager service to run as the new service account.
At this point SRM should run again. Once it does, repeat the above steps on your other SRM server.
Patching dns_updates.pl
Here is a patch file I created with the two changes I made to VMware's dns_updates.pl
53,57c53
< #
< # Dusty - if you put single quotes around double quotes, you'll be able to use a command lines with spaces
< #
< use constant COMMAND_DNS => '"c:\\program Files (x86)\\VMware\\VMware vCenter Site Recovery Manager\\scripts\\callouts\\dns_update.cmd"';
< #use constant COMMAND_DNS => "dns_update.cmd";
---
> use constant COMMAND_DNS => "dns_update.cmd";
615,621d610
< #
< # Dusty - once you process a record, clear the stack.
< # without this modification, the script won't properly handle multiple servers in a single file
< # and the result is random DNS changes.
< #
< @commonLines = ();
< @idaLines = ();
Patching dns_update.cmd
Here is a patch file I created with the multiple changes I made to VMware's dns_update.cmd
36,40c36
< #
< # Dusty - a non-domain admin has trouble executing c:\windows\system32\dnscmd.exe on a Windows 2008 server
< # Copy it to another directory and call it directly instead
< #
< SET DNSCMD="c:\program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\dnscmd.exe"
---
> SET DNSCMD=dnscmd.exe
285,288c281
< #
< # Dusty - don't bother modifying path, there's no need - call each executable directly with a full path instead
< #
< rem SET PATH=%PATH%;%TOOLSPATH%
---
> SET PATH=%PATH%;%TOOLSPATH%
301,303d293
< #
< # VMware kb1031095
< #
305,308c295,298
< IF [%VMware_RecoveryMode%] == ["test"] (
< IF %SKIPMODE% == 1 SET RUN=0
< ) ELSE IF [%VMware_RecoveryMode%] == ["recovery"] (
< IF %SKIPMODE% == 2 SET RUN=0
---
> IF [%VMware_RecoveryMode%] == [test] (
> IF SKIPMODE == 2 SET RUN=0
> ) ELSE IF [%VMware_RecoveryMode%] == [recovery] (
> IF SKIPMODE == 1 SET RUN=0
310,314c300
< #
< # Dusty - this change lets us call dns_update.cmd from the command line in test mode without having to run it from SRM itself.
< #
< ) ELSE IF %SKIPMODE% == 1 SET RUN=0
<
---
> )
316,320c302
< #
< # Dusty - display skipmode for debugging
< #
< ECHO Skipmode "%SKIPMODE%" Mode "%VMware_RecoveryMode%" is skipped. Exiting...
< GOTO :END
---
> ECHO Mode "%VMware_RecoveryMode%" is skipped. Exiting...
356,363c338
< #
< # Dusty - As delivered, dns_update.cmd will delete an existing A record at the failover IP address
< # before it creates the same exact same A record.
< # This isn't what we want, as it won't delete an existing A record at the production IP address.
< # You will have 2 DNS records - one for your failed production site, and another for your DR site.
< # This change is for Active Directory DNS servers only.
< #
< %DNSCMD% %DNS-SERV% /RecordDelete %FORWARD-ZONE% %HOST-NAME% A /f
---
> %DNSCMD% %DNS-SERV% /RecordDelete %FORWARD-ZONE% %HOST-NAME% A %%i /f
updatedns.cmd
I created a small batch file called
C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\updatedns.cmd
to help me call VMware's scripts more easily:
@echo off
if (%1)==() goto NOPARAM
if exist "C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\%1.csv" goto OKNOW
echo invalid argument passed...
exit /b 2
goto END
:OKNOW
"C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\external\perl-5.8.8\bin\perl.exe" "C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\dns_updates.pl" --csv "C:\Program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\%1.csv" --level TRIVIA --skipmode test
goto END
:NOPARAM echo no argument passed...
exit /b 1
:END
Note that the line that calls perl (everything between OKNOW and goto END below it) is a single very long line.
Creating a CSV files with the DNS updates
Now that you've made all these changes you can run dr_ip_customizer with the -extradnsscols option create the csv file with the 9 DNS fields, then edit the file to add DNS changes.
More undocumented features:
1) While following the instructions in "Adding a DNS Update Step to a Recovery Plain" srm_dns_update.pdf, they fail to mention that if you are planning to update the DNS records for a linux server, you must specify adapter ID 0 in your csv file. If you attempt to specify an adapter other than 0, you get the error "Linux VMs cannot specify a DNS Server on an adapter with a non-zero ID."
2) These same instructions tell you that you need perl 5.8.8 installed in order to use the DNS update. What they don't tell you is that SRM installs perl for you in c:\Program files (x86)\VMware\VMware vCenter Site Recovery Manager\external\perl-5.8.8\bin\perl.exe
Once you are done with the CSV file of DNS updates, name it recoveryplan.csv and put it here:
c:\program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\recoveryplan.csv
Hint: break the CSV into multiple files to have one CSV for each recovery plan.
Adding a DNS update step to your Recovery Plan
By making the DNS update step the first step in your Recovery Plan, you can quickly test the DNS updates without modifying your storage.
Open SRM at the recovery site
Expand Site Recovery, then Recovery Plans.
Select one of the Recovery Plans.
Select the Recovery Steps tab.
Right click on the first step in the plan, then select Add Command
Enter the following command:
c:\windows\system32\cmd.exe /c "c:\program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\updatedns.cmd" recoveryplan
Recoveryplan in the line above is the name of the CSV file with the DNS updates.
Right click on the second step in the plan, then select Add Message.
Enter any text you like, then select OK to save the message.
You should now be able to run
c:\windows\system32\cmd.exe /c "c:\program Files (x86)\VMware\VMware vCenter Site Recovery Manager\scripts\callouts\updatedns.cmd" recoveryplan
from the command line to update your DNS settings. Remove the parameters "--skipmode test" from the end of the line in updatedns.cmd to allow test recoveries to update DNS. Again, be sure not do perform this test of DNS updates with production servers unless you are in a maintenance window.
Once you can run updatedns.cmd successfully from the command line, try to run it from SRM itself in a test recovery. Remember that normally a test recovery does not modify DNS settings - you can modify dns_update.cmd in the places where these patches are applied to have a test recovery make DNS updates. If you made updating DNS the first step of your recovery plan, you can stop the recovery plan test while it is waiting for you to select 'continue' on the message in step 2 after watching the results of the DNS update.