No More Data Loss with Your Integrations

I decided to vote by mail this month.

So after requesting an absentee ballot, the website for my state’s election commission showed that they mailed out the ballot on October 9. And then I waited. 

And waited – sometimes wondering whether public fears of an overloaded mail system this year might be true. Did the postal service lose my ballot in the mail?

Another Kind of Lost Mail – Data Loss

Those who build their own integrations also worry about message loss. But in these cases, the reason for the loss of information often involves an outage at an integration endpoint.

The endpoint outage could happen because of a variety of reasons, including a data-center outage, an outage in electricity, denial of service attacks, crashed systems, and systems down for upgrades. The tool stops working, so it cannot receive the data.

It’s as though the mail carrier arrives at your address with your mail for the day. But because someone stole your mailbox, the mail carrier keeps driving. And you don’t get your mail. Where is it? It might show up eventually. But in the case of an endpoint outage for your integration, a company often loses the data entirely.

A Frustrating Expense that Endangers SLA Compliance

The matter of data loss is usually a hidden cost of homegrown integrations. “No one thinks necessarily about the technology to deal with a failure. What happens if one end of the integration is not available?” says Carey Blunt, who is the Chief Architect for Service Management Tools at Fujitsu. Maintenance drives up expenses, and the total cost of ownership for the integration ends up being higher than anticipated.

When an IT team spends unexpected hours troubleshooting for data recovery, the company jeopardizes the timely delivery of services.

This was the experience of Virteva (now Crossfuze):

We had the REST integration between ServiceNow and ServiceNow. It was functional. It was not the best solution that we thought was possible at the time. We were having challenges with data drops between rest calls between instances. So we weren’t getting all the updates from our customers’ instances. Working endless hours on troubleshooting where something left one instance but never showed up at the other and understanding where that gap wasn’t and what we could do to solve that problem. And really the problem was we had no queuing functionality, right?

If I had a customer that I was integrating with to pull their data … during that window that the instance was going through that upgrade, we had no way to queue any data that needed to be sent to them.

And then send it once that upgrade was complete? We didn’t have that capability. And so that was kind of another gap for us, right? We needed it to be a little more resilient for handling of upgrades, or if there’s indeed an outage of some sort.

Matt Miller, VP of Delivery, Virteva (now Crossfuze)

Message Queuing – The Answer to Integration Data Loss

When you implement an integration solution with queuing, you can prevent data loss. This is store-and-forward technology.

In message queuing, the producer sends the message (the data) into a queue, often on a message bus. If the consumer – that is, the other endpoint – is active, the message is forwarded. If not, the message is stored until the endpoint is up again.

PayPal found that a message bus contributed to a more resilient solution:

Very core to our original problem is how resilient the setup is. Before this, if I had a problem, I would drop whole tables and re-sync and do all of that. Today with the message bus, I know what data has been sent through, what data is waiting, and where I can restart if there’s a problem somewhere, which rarely occurs…. 

Business wants to ensure that, hey, if there’s a fault happening … what is your fault tolerance? What is your disaster recovery model? And all of those things come into place. The message bus allows us to ensure that even if something goes down, it will not be lost. The data is still there. It just needs to be written to a certain section of the database.

Naveed Khawar, Manager of Service Analytics and Optimization, PayPal

Ensure the Delivery of Your Data

Eventually, my ballot arrived. I returned it, and the election website says that they received my returned ballot. All is good. Of course, the real proof that I voted is my “I Voted” sticker, right?

But if you lose unreplicated data in transit, you have two conflicting records. Until you correct the data, people at your company are doing their work with inaccurate data. The result might be bad reporting, bad customer experience, and misinformed decision-making.

By making use of a message queue, Perspectium DataSync removes the need for data endpoints to be up and functional all the time. Some of the customers described in this article are among many Perspectium customers that use DataSync to eliminate risk of data loss during possible outages at integration endpoints.