Fix For: Doubletake Server Protection - Start Failed!

If you use Doubletake disaster recovery solution in your environment, this notorious error is something you will see again and again, when you try to create and start a new protection job.

This is what the support guys say about the error.

The first is having .net 4 installed on the server.  You would need 5.2.2 to be able to work effectively with .net 4 as there are some other issues you can encounter.  If you cannot use 5.2.2 just remove .net 4.

The other is a bad route selected in the job workflow.  On the confirmation screen of the workflow you have the option to change configuration on 2-3 settings.  the top one (not sure what the section is called) but it has compression and such in it...  at the bottom you need to make sure that you select an actual IP address on the target to route to.  Sometimes by default the wizard picks 0.0.0.0 as the IP - and well that won’t work.

Doubletake Protection Job- Start Failed!

Well, he is partially correct. This indeed happens because of a bad route. But even after you delete the old job and recreate it again with the correct routing interface the job will still fail to start. After many failed attempts this is how i got it to work.

Start Fresh , If you can. If you have other healthy protection jobs running side by side  then this option will not be viable. In that case you can remove whatever files associated with the failed job instead of all jobs.

1. Delete the failed Job. 

Select the failed job from the Doubletake Recover now -> Monitor Jobs and trash it.

Stop the Double take service in the Source server. This will complete the trash job process.

2. Delete all configuration/log/temp files in the source server

In the source server go to Doubletake installation directory

C:\Program Files\Vision Solutions\Double-Take

Delete all files not related to the actual installation. If you know the installation date of the application, then remove all other files created after that installation date. 

Go to C:\Program Files\Vision Solutions\Double-Take\Service\Data

And delete all files in this directory

3. In the target/repository server, go to Doubletake installation directory 

In the source server go to Doubletake installation directory

C:\Program Files\Vision Solutions\Double-Take

Delete all files not related to the actual installation. If you know the installation date of the application, then remove all other files created after that installation date. If you have other healthy jobs running then, be careful not to delete files associated with those jobs.

Files to remove : <source server name>.db ,  <source server name>.shr , <source server name>.xfp

Go to C:\Program Files\Vision Solutions\Double-Take\Service\Data

And delete <Job*>.dat and  <Job*>.xml file associated with the failed job. If you are starting fresh , then delete all files in this directory

4. Remove the old, mirrored copy of the source server from the repository.

You may need to log in as the local administrator, take the ownership of the whole folder/sub folders and then delete everything.

5. Start the Doubletake service in the Source server

Once you have removed all references to the old job, create the protection job again.

In the last window of the Server protection wizard, provide the actual main IP address of the repository server.

Cross your fingers and press finish to start the job.

This may work in some instances or may not. Job will sit on "Starting Protection...” stage for some time and then fail.

If the job fails again try the steps below.

1. Stop the Doubletake service in the Source server.

2. Go to the repository and browse to the place where the failed job stored its mirrored copy of the failed job.

3. There should be a file called VRASSM.xml in the root of that directory under the source server name folder.

4. Open this file using notepad and remove all interfaces leaving just the network interface you are planning to use. This is the same interface you have selected in the last step of the protection job creation wizard.

 

Remove rouge interfaces from the VRASSM.xml file

5. Save the file and restart the Doubletake service on the target  server and then on the source server.

6. Check Doubletake Recover now console and see if the status of the job has changed from "Communication Error" to "Calculating"

Alternatively you can stop puling you hair trying to get it working and call the superb customer support service line for a consultant to have a look at the mess  you have created and fix it remotely.

Netgear ReadyNAS 104 -RN10400-100AJS - Tips and Tricks

Bought one of these NAS device 2 weeks back with 3 TBx4 Configuration.  Sales person remarked  “Jesus! That’s plenty of storage”. I smiled. Storage can and will never be plentiful.  But that is not the story I’m going to write in here. If you like you can read about my storage woes over the years from here. (Not published yet) 

Device is okay. Not too good. Hardware design has its own problems and the software it runs on (Debian Linux, 7.0 Wheezy) is not the most stable system as I have found out. But I still love it and 2 days back went and bought another one with 2TB x 4 configurations. Yes!  I did run out of space on my 1st one. 

Issues:

1. Placing the Drives in to the screw less caddies
This can be easy or can become a pain in the arse.  Out of the box, the plastic plates that holds the drives can be in stuck position and sliding them down can be challenging or next to impossible. You might not want to put too much force on these fragile slips as they can break easily. 

Fix: 
There is a small slit on the side of the metal caddy. Put a blunt/flat screwdriver through it and derail the plastic slip slowly, first from the corner. Once you de-railed and got the plastic slip out of the caddy, scrape it’s top with a sharp knife to lower it so that it slips easily in and out of the metal rail.

Once you are happy with the amount of gray plastic scraps on the table put back the slip in to caddy and test.


2. Using used/unsupported Disk Drives.

If you have inserted used/unsupported Disk Drives, System will not boot up. It will throw up "ERR: Could not mount Root RAID device" warning.

Fix:  
Restore the system to Factory default.

Steps:
- Switch off the system. I just pulled the plug because for the life of me I couldn’t find a way to switch it off.
- Power On the system and press the Reset button in the back
- Press back up Button and choose from the menu -> Factory Default
- Press the Reset Button (in the back) to choose the option.

This will reset the storage system to factory settings, erases all data, resets all defaults, and reformats the disks to X-RAID5.

Incase you are looking, the default credentials for the web console after the reset is: admin, password.


Fixes we describe from now onward require shell access to the devise. Enable SSH access to the device by going to 
System->Settings->Services->SSH in the web console.

Once enabled, Use PUTTY or other SSH client to access the devise. Login as root and the password will be the admin password you have set up during the initial setup.


3. Sometimes External USB drives will not mount. Even if they got mounted, their contents will not show up.

Fix:
SSH in to the device and manually mount them. Once mounted, they will show up in the web interface.

Example Command: mount -t ntfs /dev/sde1 /media/USB_HDD_1

4. Bulk Transfer data to the devise. 
Transferring contents from a  high capacity external drive to the NAS over the network will take weeks and might fry your switch.  Not to mention that the web interface is a pain to use when it comes to moving files around.

Fix:
Use terminal to copy files across.

External USB devices are mounted in /media
Main NAS storage is located in /data

Copy files across these locations using Linux cp command.

Example Command: cp -R /media/USB_HDD_1/MyFolder /data/NASStorage/ 


Once copied, you need to reset permissions. Otherwise you will not be able to do any files/folder operation due to lack of permissions. 

Example Command: chown -r Tiraj:Creators /data/NASStorage/MyFolder


5. Moving Contents from one NAS devise to the other.

In this case you will have to use the network. Use SCP command to directly transfer files from one devise to the other with out involving a PC in the middle. 

Example Command:  scp -r /media/USB_HDD_1/MyFolder root@NORIKO:/data/NASStorage/ 


You will not be able to get Gigabyte LAN speeds with the supplied LAN cable. They are susceptible to too much interference thus degrading the speed. To get the maximum thru output, use short, Cat6e, triple Shielded LAN cables.

Happy NASing.

Expanding VMware ESXi Guest Virtual Disk Drives

We had few ESXi hosted guest servers that were running out of disk space at an alarming rate.  Here is the simplest way to  expand the drive capacity , Including C (System) drive.

Applies to:
VMware ESXi , 5.0 , 5.1
Windows Server 2008 , 2008 R2

Steps:
1. Shut down the Guest operating system.
2. In vSphere client , go to virtual machine properties and increase the size of the disks. Provide new values in the Provisioned Size: field. (Image 1)
3. Save changes and power on the Guest virtual machine.
4. Go to disk management in the host machine. You will see newly added space as "Unallocated"
5. Right-click on the partition and select "Expand Volume" (Image 2)

Image 1 - Virtual Machine Disk Properties

Image 2 -Disk Management , Expand Volume

Update Apache and other softwares to their latest versions - Debian

Debian 6.0.5  comes with Apache 2.2.16. Using the official software repository it is not possible to upgrade this to the latest version.

Nessus report multiple vanrabilities in this quite old version of the httpd software and IMHO it is best to upgrade, if you have been given the responsibility of ensuring  the security of a front facing web server.

To make the upgrade possible, add following repository to your software sources. This time apt-get install will work.

Debain Software Source:
deb http://http.us.debian.org/debian/ testing main contrib non-free

Scan again, everything seems to be under control... at least of today. Yes green is good. Green is your best friend. This is actually no briner. But it took me sometime to find out the software sources  and hope someone else doesn't need to dig too far for it.

Here are some more sources for Debian. Its good not to use unstable repositories on production servers. :)

Some more Debain Software Sources:

deb http://security.debian.org/ stable/updates main contrib non-free
deb http://http.us.debian.org/debian/ stable main contrib non-free
deb http://http.us.debian.org/debian/ testing main contrib non-free
deb http://http.us.debian.org/debian/ unstable main contrib non-free

Configuring vsftpd on Debian

Vsftpd is a robust and secure FTP server for Linux systems. In here I will give step by step instructions on configuring it to use SSL for secure communication  and to authenticate via a user file.

I was never able to get it working with the pam/MySQL authentication system and finally had to revert to this method which in my humble opinion is not at all bad for a small server system with limited number of users.

Step 1 : install vsftpd

#sudo apt-get install vsftpd

Step 2: create a Certificate for secure communication

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 

 -keyout /etc/ssl/private/vsftpd.pem -out /etc/ssl/private/vsftpd.pem

Openssl will take you through some basic question and answer rounds and once you have given all the  necessary details a certificate file will be created under the directory/etc/ssl/private/

Step 3: Create a directory structure for FTP uploads with a home directory for each user

This is going to be the FTP upload directory for the user tiraj.adikari. If you want to use vsftpd with apache , then you should crate this folder with   in /var/www/ . Thats the default directory for apache to store web contents.

Ex:  

#sudo mkdir /usr/ftp/tiraj.adikari

Step 4: Edit the /etc/shells file 

You need to edit this file and put a dummy shell in it for all FTP users to use. This way they will not be able to SSH in to the actual server using their credentials.

#vi /etc/shells

Add this line at the end

/usr/sbin/nologin

Step 5: Create the FTP users. One for each user who need access to the server

#sudo useradd -d /usr/ftp/tiraj.adikari -s /usr/sbin/nologin tiraj.adikari

Change the password for the new user. 

#passwd tiraj.adikari

Step 6: Create a new group and add all new ftp user to this seperate group

Lets create the group first

# addgroup ftpusers

Lets add tiraj.adikari to this group

# usermod -G ftpusers tiraj.adikari

Step 7: Create the vsftpd.userlist file to store the names of all users who need access to the FTP server.

This is the place where you tell vsftpd to check for valid ftp users.

#vi /etc/vsftpd.userlist

Add all the users you created in step 5 in to this file. Each username shoudl be in a seperate line. 

Step 8 : Edit the vsftpd.conf file

This file is located in  /etc/vsftpd.conf

Open the configuration file in vi or gedit

# vi /etc/vsftpd.conf

Set the configuration values as shown below.

# General settings

anonymous_enable=NO

local_enable=YES

write_enable=YES

connect_from_port_20=YES

chroot_local_user=YES

local_umask=022

#Authentication settings

userlist_file=/etc/vsftpd.userlist

userlist_enable=YES

userlist_deny=NO

#Secure communication settings

ssl_enable=YES

allow_anon_ssl=NO

force_local_data_ssl=NO

force_local_logins_ssl=NO

ssl_tlsv1=YES

ssl_sslv2=NO

ssl_sslv3=NO

rsa_cert_file=/etc/vsftpd/vsftpd.pem

Step 9: Restart the vsftpd service for the changes to take effect

service vsftpd restart

Trouble shooting tips:

If  vsftpd refuse to start , you can check whats causing it by manually executing the vsftpd executable and checking the output. go to /usr/sbin and execute the vsftpd executable manually 

ex: 

#./vsftpd

This will output any errors vsftp will encounter during its startup process to the terminal. Rest is up to you.

References:

• How to install and configure vsftpd 
• Configuring vsftpd for secure connections (TLS/SSL/SFTP) - VPSLink Wiki

Manage Firefox via Group Policy

Actually this heading is misleading. There is no proper easy to use  Active Directory Group Policy Object for Firefox yet.  This one shows a  a simple solution that will propogate firefox settings in to client machines via a logon script written in VB Script. Compared to some solutions this one is very simple and can be completed with in a very short time.

When a user logged in , logon scrpt given below will copy a file called user.js to that user's firefox profile folder.  User Firefox profile folder is usually located under :  C:\Documents and Settings\<username>\Application Data\Mozilla\Firefox\Profiles

You can include many default settings in this user.js file such as user home page , proxy settings and switch off or set annoying settings such as checking for default browser etc    ... that you would like to deploy in to multiple machines in the domain. Syntax of the user.js file is exactly similar to the prefs.js file that lives in the firefox profile folder.

Difference between these two files is that settings in the user.js file take precedence over the settings in the  prefs.js file and user.js file can contain subset of preferences stored in the  prefs.js file.
Here's how its done.

1. Create the user.js file with all the necessary settings you would like to be propagated in to client machines. Sample user.js file is shown below.

# Mozilla User Preferences
/* Do not edit this file.
 *
 * If you make changes to this file while the application is running,
 * the changes will be overwritten when the application exits.
 *
 * To make a manual change to preferences, you can visit the URL about:config
 * For more information, see http://www.mozilla.org/unix/customizing.html#prefs
 */
user_pref("browser.startup.homepage", "http://www.mycompany.com");
user_pref("browser.shell.checkDefaultBrowser", false);

2. Create a VB Script file (Ex: MyFile.vbs ) with following code in it  

Option Explicit
dim oShell
dim LastDate
dim filesys
dim profilePath
dim latestProfileFolder
Dim fso
dim fldr
dim MainFolder
dim sourceFile
dim targetFile
dim boolOverWrite

Set oShell = CreateObject("WScript.Shell")
Set fso = CreateObject("Scripting.FileSystemObject")

profilePath = oShell.ExpandEnvironmentStrings("%APPDATA%") & "\Mozilla\Firefox\Profiles"
' If firefox directory doesnt exist , then gracefully exit
If fso.FolderExists(profilePath) Then
Set MainFolder = fso.GetFolder(profilePath)
For Each fldr In MainFolder.SubFolders
    If fldr.DateLastModified > LastDate Or IsNull(LastDate) Then
        latestProfileFolder = fldr.Name
        LastDate = fldr.DateLastModified
    End If
Next

If latestProfileFolder <> "" Then
  profilePath = profilePath & "\" & latestProfileFolder
End If

sourceFile = "\\<Your Server Share>\Firefox\user.js"
targetFile = profilePath & "\user.js"
'WScript.Echo targetFile 
If fso.FileExists(targetFile ) Then
boolOverWrite = vbTrue
fso.CopyFile sourceFile, targetFile, boolOverWrite
Else
boolOverWrite = vbFalse
fso.CopyFile sourceFile, targetFile, boolOverWrite
End If
End If

3. Copy both files in to a shared location that has read access to all domain users.

4. Setup a group policy object so that this vbscript runs during the user logon

Refs:
Prefs.js file - MozillaZine Knowledge Base

How to find the RSA Server Fingerprint

SSH server's key is the key you see the fingerprint for when you connect to a different server for the first time. This key's identity is used to make sure you are logging in to the SSH you intend to use. 

The keys a SSH server uses to identify itself when you login to it are located in /etc/ssh/ and are named something like ssh_host_rsa_key. 

By default, ssh-keygen will create a key for the current user, which, by default, will be stored in ~/.ssh. The format of a user key and a server key is the same; the difference is where they are placed . When you install the openssh-server package, it automatically generates keys for the server to use.  

To see the fingerprint of the SSH server's RSA key, run the command : sudo ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.