EnduraData story: why file replication and synchronization?

aRoot data synchronization

Note: A reprint of an article from 2013 by SJO.

“For every complex problem there is a simple solution and it is wrong” — attributed to H.L. Mencken

EnduraData file replication topologies

EDpCloud file replication topologies

– Why are we here? Why file synchronization and replication

The reasons for founding EnduraData were professional and personal – aligning with the desire to help solve complex and  challenging problems – often based in human behavior. As many of us have discovered, there are no simple, effective solutions to complex problems.

Read on to find how EnduraData got into file synchronization and file replication …

Making complexity easier to manage

One reason for starting the company was to provide a better way for organizations to manage the large amounts of data involved within a growing enterprise – enhance productivity, mitigate risk, lower total costs of ownership.

Context – brief history of complex computing

In early computing, single users often worked on single files (like documents or spreadsheets) in a sequential order. How did we share these files? Mostly manually, using FTP (file transfer protocol), RCP (Remote copy), Rdist (Remote distribution), sFTP and scp (Secure copy). Then came databases, collaboration on files (like databases, graphics, text, video, etc.),and the eed to access and distribute files to larger groups – within the network and across the enterprise (global? No problem). Today it’s the ever increasing amounts of data creation with rapidly growing data devices and channels that drives this need. How do we manage this tsunami of data – turning it into something of value?

At EnduraData, we’re finding new (and we think better) ways to effectively share content between users, sites, business processes, enterprises and stakeholders that span many geographic locations.

EDpCloud file replication

EDpCloud is a host – based replication solution that combines data distribution, remote online backup and file synchronization. We streamline the process of moving data were you need it, when you need it. Our solution works on Linux, Mac, Windows, Solaris and many other UNIX systems meaning, if your enterprise has multiple operating systems, including those listed, youcan share data seamlessly.
Data can be sent from one system to another, from many systems to a single system or multiple systems to multiple systems. Users can build simple or complex meshes(configured links) for disaster recovery, data distribution/aggregation, or for workflow automation. The number of configured links that can be built within an enterprise is almost unlimited.

 Relevant experience and expertise

A common theme at EnduraData is working to solve complex IT problems, creatively and effectively.
One example dates back to a team member’s experience working with the National Weather Service. This problem was how to accurately estimate water levels of the annual snow pack in order to predict how its melting would affect various geographies from the point of origin to points down river.
Part of the solution involved creating a computer model of the geographic terrain in a specific area, designing an aerial survey program that used nuclear detectors to estimate snow water content, conducting snow depth sampling then running the program that calculated the amount of snow moisture that would melt and travel down the watershed and the rivers.

The computer processing requirements for this program were substantial. Even using existing super computers would require weeks of exclusive processing resources.

Our team member designed a process that partitioned the number and types of processes required into manageable tasks that could be processed by multiple computers (in a master/slave design) – using replication and load balancing.
The net result was a complex process that had required more than a week of super computer processing resources was split among several data centers and completing the computations in less than 24 hours, hence saving lives and property. In addition  to saving calendar days of super computer time and lives (and delivering the information faster), this model could be applied to other complex computation projects with the National Weather Service and the Military.

Another example comes from a team member’s critical development work at a company (before EnduraData) that served NASA.
NASA used data replication software to analyze film footage of the takeoff of space shuttle Discovery in July 2005. The film  , known as the Ground Camera Ascent Imagery Project, was part of the stipulations of the Return to Flight program laid out by the Columbia Accident Investigation Board following the loss of the space shuttle Columbia on Feb. 1, 2003. Discovery’s mission in 2005 was the first manned space flight undertaken by NASA since the Columbia disaster.

An investigation of Columbia’s demise had shown that a piece of foam had broken off the shuttle at launch and damaged one of the heat-resistant tiles on the orbiter, which caused the shuttle to break apart on re-entry into the Earth’s atmosphere. To try to prevent history from repeating, high-definition film was taken from 26 different angles of Discovery’s launch, and then analysed for any anomalies that could put the flight or the crew at risk.
The replication software was used to distribute film files and images of the Discovery launch to Kennedy Space Center in Fla., Johnson Space Center in Texas and Marshall Space Flight Center in Ala.

The 26 camera feeds produced massive files; the smallest was 6 gigabytes in size.  Moreover, they needed to go through a WAN and firewall built to NASA security standards. Finally, the product needed to fit into a stingy budget — otherwise NASA would have probably  have gone with a hardware-based appliance, which proved too pricey.

The product was the only software-based replication product that met the many specialized aspects of NASA’s application: it fit the security requirements, worked with its Solaris operating system and met the minimum performance criterion, which was that 14 massive  video files be replicated perfectly to the three locations within the first hour after Discovery’s launch.

EnduraData is formed

EnduraData was founded in 2006 to deliver improved tools for organizations that need a better way to share to distribute large amounts of data and eventually to protect it.

Problems our software helps solve

  • Data security
  • Data migration
  • Data movement and file sync
  • Workflow optimization
  • Data control and access
  • Business continuity and disaster recovery,
  • Data distribution
  • Client privacy
  • Regulatory compliance and the need to acquire more hardware
  • Software and personnel (IT spend).

Example

Problem: A growing company with multiple branches faces the problem of sharing data, including reports that are generated several times a day. Because the company made several acquisitions over time, its IT infrastructure is made up of several technologies
from various vendors like Microsoft, Sun Microsystems, Red Hat, IBM and Apple. The company needs to have the data in a central location and to allow the various branches access to the same data.

Several issues compound the problem:

Multiple hardware and software systems vendors’ require additional solutions to deal with data distribution.
No one single vendor can deal with distributing data to/from all platforms used. If a disaster hits one or more branches, significant time and effort will be lost. Every day, IT staff spends a significant amount of time changing scripts and making manual changes in order to deal with data distribution.
The vendors solutions do not inter-operate with each other, making it necessary to purchase solutions from all vendors. Once in a while the network becomes congested or interrupted and data may be delayed or may need to be retransmitted.

Therefore the company will have:

  • Major time loss by IT staff
  • Islands and silos within the company that will not be able to access or share data
  • Significant duplication of efforts and costs (c) significant management overhead
  • Redundant efforts to learn the solutions from multiple vendors and
    Significant productivity issues stemming from data unavailability
  • Risk of non-compliance with Service Level Agreements or Regulations.

Using the EDpCloud  cross platform data distribution, the company will now be able to

1. Use a single solution to address the problems cited above since it is compatible between all platforms (Windows, Mac, Linux, and Solaris) from various vendors.
Windows machines will be able to distribute data to Linux and vice versa. The same is valid for all other platforms.
2. Consolidate data from all branches into a single location using controlled access.  Distribute data from all branches and offices to all other branches and offices using controlled access.
4. Use one software solution to the job instead of using software from multiple vendors with each one vendor solving the problem only for their own platform.
5. The learning curve of using a single unified solution will not only solve the data distribution problem, but it will result in significant reductions in total cost of ownership.
6. The company will reduce maintenance and management costs.
7. If the company needs access to source code to add new features or for further deep integration with the work flow, EnduraData is the only company that will license the source code.

Our mission

We solve IT and business problems. Our focus is to serve companies that move and process large amounts of data; storage and data management.

We strive to build strong client relationships by treating clients as collaborative partners working towards mutual business success.
At the same time, we want to create a company that’s a rewarding place to work; where we provide meaningful projects – software development as an ongoing source of solutions.

Why work with us – how are we different (better)?

What makes our software unique is that we offer secure data distribution and file synchronization/replication and data backup/archiving that works on most hardware and most operation systems. In addition, we deliver a level of security that mitigates risk in multiple categories (i.e., client privacy, data loss, managing increasing amounts of data).
Our software solution automatically delivers secure data to where it is needed, when it is needed.

Contact us; we’re easy to work with.
952.746.4160

Click her to email us.

 

Other articles:

Data protection for Mac, Windows, Linux, Solaris, AIX

 

Backup and data replication for data protection

EnduraData story: why file replication and synchronization? was last modified: April 2nd, 2018 by aRoot