Overcoming Hurdles to Integration Scalability

When dealing with automation, IT leaders are aware that stumbling at any point along the paths in the automated system will impair that whole system.

They want seamless connectivity that continues to function as the number of integration endpoints grow and as the volume of transferred data grows. Unfortunately, when it comes to the typical integrations in service management, breakdowns happen frequently, stunting growth.

What are the Hurdles to Integration Scalability?

1. System Updates or Replacements

When a system is replaced or updated, custom-built integrations to that system may need to be replaced, creating additional overhead and maintenance that impedes scalability, and slows your ability/willingness to upgrade in the first place.

2. Integration via Web Services

Integrating via web services calls is common. But doing so has several limitations: max rows per query, batch processing, memory limits, semaphore usage, incoming connections, and user accounts required.

In the past, CDW used web services for ETL (extract, transfer, load) integration work. But with web services, scalability was limited. The integration “worked okay for a little while. But those ETLs put a lot of stress on ServiceNow because of how the web services work. . . . We outgrew that after a little while” (Paul Liesse, Supervisor of Managed Service Applications, CDW).

3. Swivel-Chair Integrations

Working with a swivel chair means copying data from one source and entering it into another. Or it could mean having to log into multiple systems, switching from one to another depending on the type of work that must be done.

On a small scale or for a small startup, swivel-chair work makes a certain amount of sense. But it’s just not scalable. As the data volume increases, so does inefficiency and the risk for error.

4. Point-to-point Integrations

In this model of integration, data flows directly from system to system. Point-to-point integration starts simple, but becomes complicated and quickly turns into hard-to-manage “integration spaghetti” as it gets larger.

5. Hiring and Training Personnel

For your integrations, whether you take a swivel-chair approach, a custom-built approach, or an integration-toolkit approach, you face significant increases in integration labor as your business grows. For any model other than integration as a service, you confront staffing issues when an integration breaks, when new security or privacy rules apply, or when your trained personnel leave.

Finding a Scalable Answer

The scalable answer is a message-bus architecture, made available through integration as a service.

Message-Bus Architecture

An integration solution that makes use of a message-bus architecture is not bound by the limitations of traditional web service interfaces and does not require incoming connections to any data target or source.

With a message bus, all systems follow the same standards and can share in a consistent method of transferring data between the systems. Any new system can plug into the bus, as long as it meets the standards. The setup allows the replication of data to as many endpoints as desired with no impact or change to the endpoints. Any data source that can post to or retrieve data from the message bus can be effectively integrated to any other data source that can do the same.

Integration as a Service

Your integrations also scale easily when the integrations are implemented, monitored, and maintained as a service by your integration provider. Up front, you can know the provider’s exact costs for integration as a service, without your having to worry about finding, hiring, and training more talent for growing integration needs. The scalability of personnel is inherent in integration as a service.

When your instance updates, when your endpoint is replaced, when new security and privacy rules take effect, when personnel in your IT department move on, you can relax about the integrations that your provider is supporting. Your organization is not in the integration business. So when you rely on integration experts for your integrations, you free yourself to focus on what you do best.

Scalable Integrations in Action

Using such an integration, ServiceNow moves data from 10 production instances of ServiceNow into 4 SAP/HANA databases for analysis by Sales, Marketing, Finance, and other departments. Production data alone accounts for over 10 million transfers per day – from more than 600 individual database tables – resulting in well over 2 billion records since project inception. Normally, an integration solution of this scale would require a large team, but it is handled by just 3 people at ServiceNow.

Matt Miller, VP of Delivery at Virteva, found that integration via message bus was the scalable answer, not only for performance but also for adding endpoints. “We’re over 20-plus connections to other instances right now. And so all of those connecting to us directly – we knew that was not a scalable option, so we needed to find a solution that actually moves that onto a message bus.” For Virteva, adding another integration, one between Salesforce and ServiceNow, meant simply plugging into the existing, scalable framework. “Again, it’s just another endpoint that’s now connecting back to the . . . message bus. So it doesn’t have a material impact on my performance.”

Grow Without Fearing the Bottlenecks

If the amount of data to transfer were to increase dramatically for you, which does happen for growing companies, any bottlenecks will impede that growth. Pursuing a solution that places no limits on your current or future volume of data to be sent allows growth to continue without obstacles.