Linux real time file replication and synchronization software

aRoot file replication software, support, support tips

enduradata edpcloud large scale data movement

 

In this article, I will discuss how to configure EDpCloud Linux Real Time File Replication and Synchronization to move and sync files automatically from one or multiple Linux servers, laptops or desktops to a remote site for data protection or for other work flow purposes (The remote site can be Linux, Windows, Solaris, etc).

Download

Download EDpCloud real time data replication software and a free demo license

First run uname -a. If you have a 32 bit OS, download the 32 bit package. Otherwise download the 64 bit package

Extract the tar file and run the real time file replication install scripts

tar xvf packagename.tar.gz
cd enduradata_edpcloud
./install.sh

You will be prompted for some parameters such as the install directory. Just use the defaults for now but don’t start the file replication services/daemons yet, we will do that after we configure Linux real time file replication software.

 

cd to the edpcloud bin directory (under the install directory/edpcloud/bin) and source in the script to setup the environment (this will be done automatically later)

 . enduradata_env

Please note that once you source in the env, your paths are set and some aliases are created for you.
Install one of the Linux real file replication components:
 ./edinstallfuse.sh

If your linux-utils is older than 2.18 then touch the file below in the bin directory to make sure the correct mount is used (To find out the version of your linux-utils: run mount –version to find out the version):

 . enduradata_env
touch ${ED_BASE_DIR}/bin/has_old_mount_version 

If you had to do that, you must update your operating system.

Install the real time file replication license by copying it to:

${ED_BASE_DIR}/etc/edlicense

Configure

  • cd to the edpcloud etc directory and add the directories you want to replicate in real time to edfsmonitor.cfg
 
Example:
/data1
/data2

/data1 and /data2 entries must exist in the file system since these will be mount points.

  • cd to the edpcloud etc directory and add the top directories( you want to replicate in realtime or in scheduled mode) to the includes file (use regular expressions)
 
Example of $edpcloud/etc/includes
^/data1
^/data2

You will need to add other directories if you want to replicate them (in real time, on demand or in a scheduled manner).

  • cd to edpcloud etc directory and create eddist.cfg

Example: To replicate data in real time from any localhost to 192.168.32.50 and put all data in /media/bob/backup/%ip%”. The “%ip%”  macro will be converted at run time by the receiver to use the sender’s IP. Make sure the directory exists and that the “.” in the IP is an underscore and not a dot. You will see the receiver log complaining if the directory does not exist (see example below)

mkdir -p /media/bob/backup/192_168_32_2 #if the sender is 192.168.32.2
 <?xml version="1.0" encoding="UTF-8"?>
<config name="enduradata" password="zooo" >
    <link name="or" >
        <sender hostname="localhost" 
            alias="*" 
        />
        <receiver  hostname="192.168.32.50" 
                storepath="/media/bob/backup/%ip%"  
        />
    </link>

</config>

Your server file replication configuration is done.

  • Start edpcloud services now using the following command:
 
cd $installdir/edpcloud/bin
. edpcloud.sh startall

Verify that edpcloud file replication software is running

 
edstat
df

edstat will show you the status, the number of files in the journal, the bytes on the move, etc.

Examine the output of df and you will see that edpcloud mounted both /data1 and /data2.

  • If for any reason edstat is showing any errors then you need to
    examine eddist.log under edpcloud logs directory (other useful logs are edfsmonitor*log, ed_receiver*log, ed_sender*log).
  • Test that file replication is taking place in real time and that your local data under /data1 and /data2 is being synchronized as it changes on the localhost to the remote server:
  • Now go to /data1 and touch a file or write to it, rename it , etc and you will see your changes replicated to the remote host. Please note that deletes are not propagated by default.

Deletes on the remote host

If you want deletes to be propagated then add the deletes=”1″ to the configuration as shown below.

 <?xml version="1.0" encoding="UTF-8"?>
<config name="enduradata" password="zooo" deletes="1" >
    <link name="or" >
        <sender hostname="localhost" 
            alias="*" 
        />
        <receiver  hostname="192.168.32.50" 
                storepath="/media/bob/backup/%ip%"  
        />
    </link>

</config>

You will need to perform the initial synchronization to copy the local files and directories first:

edq -l or -r 192.168.32.50 -n /data1

edq -l or -r 192.168.32.50 -n /data2

You will need to perform the initial sync only once and after that all file changes on /data1 and /data2 on the localhost will be sent to the remote host in real time.

If you want to synchronize and replication multiple servers to the same remote host, simply install and drop the files you created in the edpcloud/etc directory on each of the servers, run edpcloud.sh restartall and you are on your way to consolidating your backups, aggregating your content or whatever you want to achieve (Don’t forget to get the right package and to run the following command (IF and only if your mount version is older than 2.18)

touch ${ED_BASE_DIR}/bin/has_old_mount_version

Your Linux real time file replication and synchronization software is installed, configured and is now keeping your many servers in sync to automate your business processes and your data protection using file replication for online backup. Make sure you visit again and read about creating file archives.

Next time we will talk about archiving file changes. Until then, be safe but above all protect your data.

 

If you like this article, please share it. We would also love to hear from you.

–a

 

Contact support for free help

Additional articles you may find interesting:

Linux real time file replication and synchronization software was last modified: April 2nd, 2018 by aRoot