The LINK®: Technical Aspects

With the ubiquity of powerful software applications today, integrating applications by automating data flow is no longer the business differentiator it once was. Today’s integration challenge is not in merely automating data population in multiple systems, but systematically leveraging the capabilities of each system to create cascading inputs that automate all facets of your data. Using templates and rules to not just move data, but expand and extend your integration into all areas of downstream systems automatically; that is your business differentiator. How well you strategically integrate your entire systems (not just some raw data) to overcome the need for manual follow-up differentiates you from your competition and creates a competitive advantage.

Given today’s competitive marketplace, companies cannot afford to perform redundant tasks, or to miss out on the leverage that can be provided by integrating applications with one another programmatically. As an example, how could a company afford to enter orders into three different systems in three different ways, with three different sets of rules, if a competitor receives the orders electronically and has the requisite follow-up activities (however different or complicated) cascade automatically through countless in-house systems that plan, prepare, manufacture, configure, and ship the order out the door automatically. From a business standpoint, connectivity among applications is a must today. Shop floor, inventory, accounts receivable and advanced planning applications need to communicate automatically and seamlessly so that companies can make accurate promises to customers, orders can be fulfilled, and executives can make educated decisions more quickly. The challenge today is to learn to leverage integration applications in new and insightful ways, to gain an edge over competitors who still rely on “keystroke automation”, or pursuing enterprise integration from a single software package.

Enterprise Application Integration (EAI) first started in the 1960’s as the idea of automating routine tasks in order to remove the manual labor. As needs changed and evolved, elements were added to the automation to tweak the data based on data from other sources. Over time, this simple automation evolved to encompass more than just reading data and automating inputs; it grew to involve moving data from one system to another, and finally leveraging the momentum to implement more advanced integration strategies. Interestingly, this decades long evolution of EAI mirrors today’s usage of within corporations. Companies start with simple automation needs that grow as the company grows, eventually to include full-blown EAI requirements involving multiple enterprise systems. As companies evolve, they often change their integration systems in an attempt to find solutions more and more suited to the challenges they face at each growth phase. All too often, the changing of integration systems fails to leverage existing software and simply re-implements without re-strategizing.

EAI has evolved over the years from batch processes on multiple machines, to monolithic applications centralizing all integration processes, to a Services Oriented Architecture (SOA) where the integration services are fragmented into non-local web endpoints. The SOA architecture has gone from the go-to standard for new integration projects to just another available technology. New integration projects today often use whatever technology seems most appropriate (batch processes, a single monolithic application, SOA, or a combination of these). Missing Link Technology introduced The Link® in 2004 as a new way of performing and managing integration projects through
Windows Services.

Batch Processes: EAI was initially accomplished through batch processes running in DOS (or similar). Batch processes were predominant for many years as being the quickest way to achieve results, yet relatively simple to implement changes. Unfortunately, batch processes fail in just about every other measure. They are slow, not scalable, cannot easily be distributed or coordinated, and utilize older technology and a limited development language.

Applications connecting applications: EAI evolved from batch processes to applications in the mid-nineties when integration “middleware” applications began to appear and companies began converting their batch processes to internal applications. Applications had advantages over batch processes as they utilized new technologies, could run faster, improved interfaces, easier to manage, and could be developed in practically any programming language available.

The problem with applications connecting applications: In this scenario, the “middleware” application contains all the rules and functions necessary to achieve the integration of two (or more) systems coded into it. Initially, this was relatively straightforward but over time the rules, definitions, and functions contained within the middleware become very complex, until it is inevitably more complex than the actual applications being integrated. This ever-growing complexity makes them increasingly difficult to extend, troubleshoot, and operate. The middleware applications themselves need dedicated managers and other formal resource allocations in order to continue functioning, and as any application gets more complex, it inevitably becomes less stable. Middleware applications can grow to be management nightmare, and the business communities that rely upon connectivity throughout their enterprise have a need for something less complex, and more stable.

SOA methodologies revolutionized the systems integration industry in the early 2000’s.  The problem is, they started out with some initial shortcomings and have evolved to conquer new business problems and not to address their own inherent shortcomings.

Services Oriented Architecture – SOA: In the early 2000’s, Services-Oriented Architecture (SOA) entered the integration lexicon. SOA was a break from the old application paradigm as it moved many of the integration tools, definitions, and processes into de-centralized distributed web services. This approach also required an application to coordinate calls to the web services, as the web services themselves are not autonomous and must be invoked explicitly. Web services simply respond to requests, and thus they still require a management application to coordinate and consolidate the requests. The beauty of SOA integration is that the business rules and definitions required for integration can be stored anywhere on the network (or Internet) and are not required to be contained within a single program; this allows applications to be more agile and less complex. Since the rules can be distributed, SOA architectures are also much more scalable and manageable than single-application integration solutions. This made SOA the go-to architecture of the time.

The problem with the traditional SOA: Even through SOA became the go-to architecture for integration solutions, it still had a few shortcomings. The web services themselves could not perform any work unless they were called upon by an application. They are not autonomous. The application that calls the web services still grows in complexity as each new service is added. Although the services can be located anywhere, they work best when close to the application coordinator, business rules reservoir or other data element; often distant services experience unacceptable lag times. Web services are also not easily moved from one machine host to another, which is often needed for load balancing during scale-out of the solution. And a common complaint from developers is an SOA using web services can be difficult to debug.

A Better Way: SOA over Windows® services instead of web services.  
The LINK® entered the integration marketplace as a formal product in 2004. It is built on the idea of using Windows services for integration. It uses a methodology that leverages the advantages of both applications and the SOA. The LINK® services have the benefit of web services, but without requiring a management/coordinator application in order to run, as needed by the traditional SOA.

The LINK® services:

Is deployed with a mouse click to any Windows machine in the enterprise. No IIS or other pre-installed software is necessary.

Will work autonomously without the need to be called or coordinated by an application.

Is easily monitored for throughput versus processor load, and moved easily to different machines.

Has (detached) user interfaces and can be monitored in real-time without consuming processing cycles on the machine hosting the service.

Is be easily debugged in real-time, a task that is incredibly difficult with traditional Windows services.

Runs as true Windows services, giving them special priorities and privileges in the Windows operating system.

At the same time, The LINK® services have many of the benefits of application-based integrations. The LINK® does this by separating the business rules and work from the user interface. The business rules and work are included in the services themselves. The user interface is separate, required only for initial configuration, and allows for ongoing activities such as service management, deployment, monitoring, reporting, tuning, etc. This separation allows The LINK® services to leverage the advantages of both applications and services, while simultaneously eliminating the traditional shortcomings of each technology.

Missing Link Technology has focused on developing strategic EAI solutions since 2001. Strategic EAI implementations show their strength during data movements from one system to another by introducing templates, transformations, or mechanisms that can turn one transaction into many and create many outputs from fewer inputs. Data from within each transaction is used to identify templates or rule sets, and each individual rule set carries assumptions that allow still more data fields to be populated automatically. This methodology can create multiple transactions from a single source and large differences in downstream data populations according to the data in the source transaction and assumptions made in the templates applied. In doing this, the implementation becomes strategic, achieving results that are more than the sum of their information-moving-parts.

Integrations are by their nature very complicated. They pose a difficult problem with no one clear resolution. As a result, the integration problem is nearly always broken down into a series of smaller sub tasks that, taken together, form the desired solution. Normally this componentization of the larger problem is done as a mental aid to help to describe the problem and render it more readily solvable. The LINK takes this notion of componentization a step beyond a simple mental aid to the enforcement of componentized solution as part of a distributed services integration model. In this model, each service does one thing (and does it well). This componentized one-task-per-service approach is the opposite of the more common monolithic approach to integration that uses single large, often hosted applications to handle as many tasks as possible.

The LINK® distributed services integration approach is similar to conventional SOA models in that both utilize distributed services. But unlike a traditional SOA that relies on web services to be invoked by a controlling application, The LINK services are themselves capable of both management and implementation. In other words, you don’t have to call a The LINK service from an application to perform a task; The LINK services operate autonomously, without the need for any applications to coordinate tasks. The LINK allows companies to leverage distributed services into a strategic, coordinated workflow. This, coupled with MLT’s focus on strategic integrations, allows The LINK to be utilized as more than just the sum of their parts.

But utilizing an SOA also implies that the services-oriented approach doesn’t end with the integration system, but extends throughout other facets of the enterprise including components of core applications themselves, not just middleware. The LINK not only works within an SOA but is able to invoke and coordinate web services, something an application would normally be required to do.

With the Missing Link Technology model and The LINK, you can have an SOA controlled entirely by services, with no applications required to integrate your enterprise applications.

Since The LINK® is made up of Windows services that can be distributed over multiple machines or even across WANs, it presents new integration options not available to other systems that rely on a single monolithic application to provide their integration needs. Services can be moved to other machines to facilitate capacity and scalability concerns, but more commonly they are moved so they may reside closest to the system they are monitoring or receiving data inputs.

Integrating disparate systems over a WAN can be a challenge with conventional integration models. If you have systems that are very far apart, where do you run the integration application? If it is run at the monitoring site, then transferring data to the target system imposes a large lag time. If it is run close to the target system, it is difficult to monitor the source system due to lag time issues at the other end. All too often the solution is to purchase multiple copies of the application to run one copy at each site. With The LINK this scenario is handled as part of its integration model; simply place one The LINK service close to the source system and another The LINK service close to the target system. Then the entire issue of dealing with remote systems goes away and the only remaining issue is that of passing data over a WAN, something for which a wide variety of technologies are available (SOAP/XML, Web services, FTP, HTTP, etc.)

Below is an example of integrating ERP systems residing at different physical locations with a corporate SAP system located elsewhere.

With integration projects done the old way, using the monolithic model, communications and coordination were accomplished by using program variables, or simply placing one function after another, thereby ensuring that the function listed first in code will run first, and the function listed second in code will run second, etc. Under SOA, services run independent of each other, and potentially on separate machines. Thus, under SOA, communications and coordination between various services requires its own mechanisms. The Link® accommodates this through its tight integration with the Microsoft Message Queue (MSMQ).

The backbone of any SOA integration system is in the ability to leverage individual service actions by coordinating them with other services. This can be done through inter-service messaging using message queues like MSMQ as well as other third-party message queues accessible through the .NET development environment. All The Link® services are automatically pre-registered publishers and subscribers to MSMQ, giving them the ability to receive messages from other services with no additional coding required. Message receipts are handled on a dedicated thread and sending messages can be done with a single line of code.

In addition to the ability to send and receive messages, the MSMQ has features that allow for blocking to take place; that is waiting for a separate service to complete its processing prior to continuing. This can be useful in enforcing specific workflows, where actions must take place in a serialized fashion. This and other MSMQ features allow The Link® to be easily and powerfully chained together to produce a desired result that utilizes multiple distributed services.

Support for message queues goes a long way toward solving problems with service communications and transactional hand-offs, but sometimes the ability to interact with other services in ways other than simple messaging is required. Such as the need to interrogate services to discover or modify: metadata, errors, log files, transactions, configurations, etc. To do this, more than just support for a message queue is needed. The LINK® allows for this sort of cross-service interaction with two mechanisms, one is The LINK API, and the other is a well-designed table model that lends itself to open access.

The LINK API

The LINK comes with an extensive API that allows for programmatic integration with any Virtual Studio language. The API provides public access to all deployed services as containerized objects utilizing a model that allows for intuitive service navigation and support for use of For Each loops and LINQ access/manipulation.

‘ Simplified access to common functions using a single line of code

' Access the database
Dim lastTrans As Date = MLTDB.RunSQLReturnDate("Select Top [DateTime] From MyTable Order by [DateTime] Desc;")
MLTDB.RunSQLExecuteNoReturn(String.Format("Insert into MyTable (Field1, Field2, Field3 Values {0}, {1}, {2}", 1, 2, 3))

' Send email
MLTEmailSender.SendEmail("[email protected]", "My Subject", "My Body", "My Attachments")

' Access FTP files
Dim folderExists As Boolean = MLTFTP.FTPFolderExists("Public/Interfaces/XMLFiles", "ftp.myco.com")
Dim folderContents() As String = MLTFTP.FTPGetDirectoryContents("Public/Interfaces/XMLFiles", "ftp.myco.com")

' Access other service instances
Dim timeWatcher1Enabled As Boolean = MLTInstances("TimeWatcher1").Configuration.GetBoolean("ProcessingEnable")
For Each instance As MissingLinkService.Instance In MLTInstances
    Dim configFile As String = instance.ConfigurationXMLFile.FilePath()
    Dim exePath As String = instance.ConfigurationMaster.ExecutablePath
Next

' Access data from your watcher (table Watcher)
For Each dr As System.Data.DataRow In MLTWatcherTable.DataTable.Rows
    MLTTrace.Log(String.Format("First field is {0}", dr(0)))
    MLTTrace.Log(String.Format("Second field is {0}", dr(1)))
Next

' Access data from your watcher (directory Watcher, event log watcher)
Dim fileContents As String = MLTWatcherDirectory.CurrentFile.Contents
Dim eventData As String = MLTWatcherEventLog.CurrentEvent.Data()

(Sample screen-shot of The LINK API Code Reference documentation)

Interface Tables

In addition to The LINK API, the service data tables (master, transaction, message, log, error, and more) are available in The Link® database (SQL Server) to other services and/or applications openly and do not require referencing The Link® API. These tables facilitate a very generic and extensible inter-process communications model. They exist, in part, to facilitate information sharing between unrelated processes where API usage is either not possible, or not desirable. This generic inter-process communications mechanism provides an extensible communications model that allows applications and services not related to The LINK to easily integrate into the integration landscape.

After more than three years of pre-market development including internal and client testing, The LINK® was released in 2004 with mature technologies and methodologies already honed over several years. The Link® event engine uses a standardized technology applied in multiple ways to facilitate a number of different ‘Watchers’ of various event types.

Real-Time Event Engine

At the heart of each The Link® Watcher is a proprietary event engine, built with base elements that intrinsically support event-agnosticism and real-time event monitoring. This event engine is inherited by each event watcher type and tailored to support the specific event needs of each type. It spawns three main threads as a part of its processing methodology:

1)     Event thread.  Unlike web services, Windows services are always running and available and able to monitor activities without the need to be called upon explicitly.  The event thread acts as the manager and coordinator for all service actions and events that occur on an ongoing basis.

2)     Messaging thread.  The Link® services are tightly integrated with MSMQ and each service automatically inherits a dedicated MSMQ thread to ensure immediate and simplified access to Microsoft MSMQ.

3)     Worker thread.  Each Watcher type has its own worker thread for performing its specific duties.

Directory Watcher

Uses events built into the Windows file system in combination with other methodologies to ensure that newly created files are noticed with minimal I/O activity and that no file creation event is missed; even if the service is not running when the file-creation event occurs.

Dynamics Watcher

Uses X++ calls into Dynamics AX to monitor a Dynamics AX database in memory in real-time. As a result, the Dynamics watcher never uses disk I/O to monitor Dynamics AX tables or read new records. Like all services, the Dynamics Watcher has accommodations to ensure that no new record addition is missed; even if the service is not running when the record is added.

Email Watcher

Uses a proprietary library for attaching to an email account in a manner that is optimized for ongoing monitoring.

Event Log Watcher

Windows exposed API methods are focused on processing event log records as part of batch processing. Fortunately, The Link®’s proprietary process is optimized to monitor the Windows event logs in real-time, at the same time ensuring that events are never duplicated, and never missed; even if the service is not running when the event occurs.

File Watcher

Uses events built into the Windows file system in combination with other methodologies to ensure that changes to files are noticed with minimal I/O activity and that no file additions are missed; even if the service is not running when the file-change event occurs.

Table Watcher

Uses specially optimized technology to ensure that records added, changed, or deletedare recognized in real time (within milliseconds). The specialized technology leaves the target database untouched, by avoiding use of triggers, or the SQL-CLR. The Link® leverages SQL Server memory objects to recognize in real-time when new records are added without utilizing disk I/O actions during periods of inactivity, when no new records are being processed.

TCPIP Watcher

Uses a proprietary library to enable access to raw TCPIP sockets while supporting terminal standards that allow connection to mobile devices of all eras. With the TCPIP watcher you can connect to everything from circa 1960’s “green screen” terminals to the latest smart phones.

Time Watcher

Uses a simple timer to trigger custom code at regular intervals. The Time Watcher service is designed to implement monitoring (watching) methodologies in service code without having to choose a built-in watcher type that might not suit the specific monitoring needs of a particular integration pathway.

Web Page Watcher

Uses HTTP POST/GET technologies to access web pages, including the ability to post information (such as credentials and other details) in order to access context-specific areas of web sites.

In The LINK® integration model, each integration pathway is managed by a true windows service, not an application. Normally services cannot be made robust enough to handle the needs of a complex integration. The Link® makes this possible by simplifying and componentizing each integration task into individual services. Each .NET service uses the Missing Link Service Base class, which contains a comprehensive collection of objects, properties, and methods required to facilitate The Link® model. This includes: MSMQ, database communications, versioning control, mechanisms for custom database communications, TCP/IP, email, ftp, http, XML, file-reading, Windows events and more. The base class contains everything required in order for the service to run as a Windows Service. The Link® uses Visual Studio templates to create .NET projects; the base service code is already in place and all you as a developer need to do is add your integration code to the provided methods.

“Other solutions often added hours, or even days to the length of a project.  As a direct result of their advanced development model, Missing Link Technology’s innovative The LINK software suite allows us to create reusable integration services in minutes.” Mat Jones Director, Griff & Jones Ltd.

This model keeps new code required to an absolute minimum. There is no required bootstrap code, no GUI, no management, no responding to user clicks, no additional logging, no code to manage events, no required anything other than exactly what the specific service needs to do. One hundred percent of the coding effort is on applying business rules and data transformations and not on tasks that do not have to do with the integration problem at hand. All of the code required to run the service is built into the base class and user interface code is handled through The LINK modules, such as the Service Manager. This makes for rapid development of new The LINK services with the minimal amount of code possible.

The Big Brother Service, which ships with The LINK, complements the created services by continually monitoring a number of different aspects and takes action to start/restart it if any problem situations are found:

• Service is not running (for any reason)
• Service has a memory leak
• Service has not been restarted within a specified periodic restart interval
• Service has encounter an error that you have deemed ‘fatal’
• Service is not responding in a timely manner (using Windows API for program response times)
• Service is not progressing through its events in a timely manner

All of this means that services stay running and healthy at all times without the need to write additional code to make the service highly available.

In addition, the Big Brother service also monitors each service’s log files and log records to automatically maintain a reasonable log history and at the same time preventing log files or records from using excessive disk space. Log files and records can be kept at a constant level regardless of the level of activity the services are experiencing. Just one more feature to ensure the focus is on integration problems and not on database or file log management.

The LINK® Service Manager application provides real-time statistics (metadata) on all instances in a chosen environment. The environment can range anywhere from a single The LINK Service to all The LINK services in a given enterprise. The Service Manager provides access to all information about a given The LINK Service, including:

• State
• Throughput (where applicable)
• Transaction statistics
• Transaction details
• Rules failures
• Historical data
• Detailed trace logs.
• Real-time stack listing.

The real-time stack listing is a unique feature. The LINK services publish and log their entire stack in real-time to both a file and a log table. This means there is no wondering what a given The LINK service is doing at any time. It is possible to view the stack log in real-time and see in detail what the services are doing, even if their outputs do not appear to be changing.

Support for user-defined custom commands allows interaction with services through a graphical interface, in real-time, just like an application. Custom commands are defined along with coded actions in repose to each command, in a Visual Studio service project. When the service is running, The LINK manager allows the user to issue custom commands to the service while it is running, to interact with the service in real-time. The LINK Service Manager also provides front-end controls to fully manage The LINK services well beyond the typical start/stop/pause/resume controls common to all services.

The Error Reporting service, which ships with The LINK, compliments the created services by ensuring any errors encountered in the service are reported. Errors in code are typically trapped by a try-catch block within the service, and the built-in .NET objects allow developers to log an error that will be picked-up by the Error Reporting service with one line of code. Even if the errors are not explicitly trapped (a bad practice, or sometimes just immature service code) The LINK code model will still log the error. The Error Reporting service is configured to monitor for all errors and report them via email to a predefined email recipient. The Link® ships with the source code to the Error Reporting service to modify the service to do more than just email error reports, such as error remediation or other actions.

Unique Benefits of The LINK®:

• Detailed graphical user interface for Windows services.
• Graphical interface for managing service dependencies (not available in Windows).
• Rapid application development (RAD) through a highly optimized architecture that lets programmers focus on the business problem, not the interface.
• Support for real-time user commands, just like an application.
• Sharing tolerance settings allow for customized performance levels.
• The LINK can be debugged directly in Visual Studio, without having to attach to an existing process – something not possible with other services.
• Speed Tweaker™ allows for user-customizable performance levels taking the needs of other service applications and services into account.
• Support for MLT-managed events, or user-defined events
• Built-in logging, reporting
• Built-in message queue support
• Built-in blocking and other service-coordination workflow mechanisms
• Built-in service error monitoring and service resetting.
• Built-in error reporting
• Built-in reporting tool allows user-defined reports to be created in seconds, and accessed with no additional interface required.
• Inter-service communication support
• Internal object collection accessible through an extensive API
• MLT Support from application integration and SOA specialists with decades of in-the-trenches experience with real world integration problems.

Key advantages of a Services-Oriented Architecture (SOA):

• Reusable components can be leveraged in future integrations, minimizing the need for additional code with each new integration need.
• Simplified, componentized programs are less prone to bugs found in more complicated, monolithic programs.
• Services can be extended into the supply chain or the sales chain without customized methodologies.
• Very extensible.  Can grow much more rapidly than conventional monolithic integration models.
• More open environment can be interfaced with other companies or other locations easily.

Key advantages of Windows services:

• No user logon required.  Services start automatically with the machine.
• Easy to Deploy, Manage, and Use
• Secure Connected Infrastructure
• Enterprise-Class Reliability, Availability, Scalability, and Performance
• Lower TCO Through Consolidation and the Latest Technology

Key advantages of the .NET Framework:

• Consistent programming model.
• Helps IT professionals better integrate existing systems with its native support for Web services.
• Assists with the deployment of software to both users and Web servers.
• Facilitates the development of software with improved reliability, scalability, performance, and security.
• Helps developers be more productive by:
  • Making it easier for them to reuse existing code.
  • Enabling them to more easily integrate components written in any of the more than 20 supported programming languages.
  • Helping them more easily build software for a wide range of devices using same skills and tools.

Best in class Features of The LINK

Power – The full power of Visual Studio with no new proprietary language to learn, not limited by macro/mapping interface.

Learning Curve – Since Visual Studio is software your developers already know, the learning curve is a matter of familiarizing yourself with the application interfaces. 20 hours of formal training are all most developers will need to start being productive on their own.

Scalability – Services can be deployed and re-deployed to any Windows machine in your enterprise with the click of a mouse with no new licenses to purchase.

Speed to Solution – Build services to connect to systems within minutes.

Affordability – Competitive pricing compares favorably with any other integration product in the SMB market space.

Reporting – Built-in reports for all levels from real-time technical reports, to summary and timeline reports, to pie chart reports on transaction classes, all the way up to KPI reports for managers.