SOA Service Gateway

Background:

The United States Department of Agriculture’s Food Safety and Inspective Service (FSIS) ensures the safety of food all across the United States by inspecting plants, taking and analyzing laboratory samples, and implementing many other safeguards that regulate the industry. Food Safety regulations and policies often change to keep up with new health needs. The public food safety system is designed to predict risk to help the agency improve predictability, and therefore reduce the spread, of disease.

Challenge:

The public health system has replaced or is replacing over twenty legacy application systems. Administration realized that information is shared across systems, and that analytics reports could benefit from aggregating data from these various systems. The public health system architecture was built around service oriented architecture, which implemented web services that are shared across the FSIS domain. As the engineers proceeded to implement a service oriented architecture, the tendency was to use the new SOAP 1.2 standard in order to exchange information between legacy systems and heterogeneous systems such as web sphere, SAS, and .Net. As web services became available to more platforms, the challenge became not how internal systems had to communicate, but how to communicate between government to government and industry to government systems. The fact is that not all agencies are ready, or have the ability to publish data in web service compatible form.

Solution:

In order to implement a consistent SOA, an SOA gateway was conceived, which is used as a hub to all external systems outside the FSIS web services domain. This gateway was equipped with adapters that abstract interfacing through various FDP protocols, shared file protocol, or web service protocol. In addition, it has a scheduling system that is configurable to run at intervals, and it monitors and audits all transactions, using the staging database as the back end for the web services that are exposed to the domain. This allows all applications in the SOA domain to get data from other systems using Microsoft Windows Communication Foundation Web Services. The Gateway provides a migration path for exchanging data with other agencies, where each agency provides data protocols at its own readiness. If one agency has a more sophisticated framework than another, the Gateway can analyze what is the most effective form of communication between the two.

Microsoft Tools Used:

  • Visual Studio 2008
  • .Net 3.5
  • Windows Communication Foundation
  • Windows Services
  • Enterprise Library 4.1

Design:

The SOA Gateway ties together all external systems with the SOA Domain, serving as a clearing house of all external services that are accessible (see Figure 4).  It also exposes selected internal services to other SOA domain gateways.

Figure 4: Contextual View of a USDA Agency SOA Gateway

Applications and services within the SOA Domain can consume any web service using the virtual end point concept[1].  The application designer can discover all available services based on their membership to a “Community of Interest[2].”  For web services internal to the SOA domain, those services self-register and are automatically discovered. For external data sources, the gateway provides a service wrapper as a web service so that external services are visible to the SOA domain as if they were internal services. This service wrapper encapsulates the transport protocol (e.g., FTP, MQ, MSMQ, HTTP, EDI, and TCPIP) and the message protocol (e.g., push, pull, pub-sub). To create this service wrapper functionality, the Gateway uses the following components:

  • Adapters are used to convert from one protocol to another in order to achieve transport independence (e.g., MQ to http, EDI to HTTP, etc.);
  • ETL tools are used to transform the external schema to an internally relevant representation in order to convert metadata;
  • Staging databases are used to temporarily store the latest external data in order to convert a message push to a message pull; and
  • “Listeners” are used to monitor files, databases, web services, or email in order to create events triggered by external systems.

The SOA Gateway provides the following level of service:

  • Provides a service wrapper for external services so that each appears as an internal service to other services inside the SOA Domain;
  • Provides synchronization of the registry using both publish-up and sync-down operations;
  • Allows visibility of services in other domains based on configurable access;
  • Provides adapters to abstract the transport protocols;
  • Provides a user interface to allow an administrator to easily publish services or data.  In addition, the administrator should be able to manage access to services using service groups or communities of interest; and
  • Provides high availability and speed by caching or staging external data.

 

[1] Virtual end point is a web service concept that exposes a web service independent of physical location, technology, and transport protocol

 

[2] Community of Interest is an organizational construct for working collaboratively to establish clusters of data interoperability that cross formal boundaries, and therefore must have shared vocabulary for the information exchanges.  EX: USDA agencies with common interest in CBP data could be established as a Community of Interest and share the security and data specifications needed to share the services used by this community.