On Integration – Developing Software with Integration in Mind

Share

I’ve posted two articles on the challenges of systems integration, firstly on the advantages and disadvantages of Best-of-Breed and Fully Integrated Solutions, and secondly on how to approach integration if you’ve chosen Best-of-Breed.

See Systems Integration and Reducing the Risks.

This post is concerned with how software should be developed if integration is anticipated.

integration3

You can think of it in this way:

Systems respond to EVENTS by executing software PROCESSES that update DATA.

In a single ‘fully integrated’ system the ‘complete’ response to an EVENT is contained entirely in one system and is reflected in one set of DATA. ‘Fully integrated’ systems can thus always ensure a complete response to an EVENT or no response, but are rarely left in an incomplete state resulting from partial processing. Typically, databases roll back incomplete PROCESSES, and all the consequent DATA changes, if there is a failure at any stage. It’s all or nothing, if you want it to be.

Integrating best-of-breed systems, working with more than one database, makes it more difficult to achieve this kind of integrity. The failure of a PROCESS in one system can’t easily trigger the roll-back of a PROCESS in the other.

Integrating separate systems requires therefore that you should easily be able to identify the success or failure of each part of a PROCESS and identify and control each set of DATA changes that each PROCESS effects.

For example, suppose you’re a company that distributes imported goods and you’re obsessive about calculating the margin you obtain on each product (this may be for good statutory tax reasons, or simply because you are unnaturally pedantic). The cost of the goods you sell is determined by the price you pay for them, the duties you pay on importing them and the cost of transporting them to your warehouse. You only know the true cost of each product once you’ve received the invoices that make up all of these costs. But, in the meantime, you may have shipped them.

Let’s consider an EVENT such as the arrival of an invoice for transportation. If the calculation of ‘actual’ costs is automated then your system must carry out the following PROCESSES:

  1. Invoice Registration: The invoice is booked against supplier, tax and cost accounts.
  2. Accrual Reversal: If an accrual for this cost has been made earlier it must be reversed.
  3. Stock Value Correction: The difference between accrued and actual transportation cost must be calculated and allocated against all the stock items it is related to.
  4. Cost of Goods Sold Correction: If there are items to which this difference in cost relates that have already been sold, then the appropriate portion of this difference must be allocated to a cost of goods sold account rather than to an inventory account.

Each step involves updating transaction tables, and if all these PROCESSES are occurring in a single integrated system then either they all succeed or they all fail, and the database either reflects a complete response to the EVENT or no response at all.

But if your finance system is separated from your manufacturing system then some of these PROCESSES must be carried out in one system and some in the other (and parts of some in both). It’s likely also that they will take place at different times (if there is ‘batch update’ of one system by the other rather than immediate update). It doesn’t matter for the moment which. In consequence it is possible that taken together the systems may reflect a partial response to an EVENT.

When developing systems that must be capable of handling partial failure in response to an EVENT the following approaches are wise:

  • Ideally there should be an explicit database table that identifies for each uniquely identifiable EVENT the PROCESSES that have been completed successfully and which are therefore fully reflected in the system’s database. Ideally such tables are present in the system that is sending and the system that is receiving. In reality, it is the state of the data that implies whether a PROCESS is complete, and it is rare to find an explicit table that records this separately, but it would certainly be useful. However the success of each PROCESS is recorded, the changes in data associated with each PROCESS should be easy to identify.
  • Transaction data that have been processed by an interface program must be marked as such.
  • All modifications to data that imply additional interface processes should be handled with a full audit trail. Ideally a modification to a transaction involves the insertion of both a reversal of the earlier transaction and a new transaction.
  • All transactions should have a field that identifies the order of events. Ideally this should be a date/time stamp. This enables execution of interfaces in the correct sequence, as well as the unwinding of PROCESSES if required.

If these safeguards are observed reconciliation and correction should not be too difficult.

System integration isn’t easy and there is no reason to believe it will get easier. There are still no standards for the description of data to which all system developers and integrators can refer. This is no surprise. Data are not objective objects which can be explored and described without reference to purpose, context and culture, and they never will be. They are not like the objects of material science; they are human artefacts.

Over the last decades I have witnessed a number of false dawns with regard to ‘easier integration’. These usually turn out to be useful advances in technology but rarely make functional integration any easier.

Two weeks ago I read an article on Data-centre Software in The Economist (Sept 19th 2015). That’s what made me think about integration issues. The article mentions Docker, a successful startup that’s making it easier and more efficient to move applications from one cloud to another. Laudable, clever technology, but, as far as I can tell, it represents no progress in the more difficult task of integrating business processes. Those of us who do such work can be assured of work for many decades to come.

On Systems Integration – What’s the problem?

Share

Integrating business systems can be a nightmare, and we who talk confidently of ‘seamless integration’ during sales presentations know this well. The fact is that it’s often a difficult and complex task, and advances in technology have done little to make the task easier. Rather, as systems have become more complex, reaching us wherever we are on devices of many different kinds, integration has become more complex too.

XML, web services, BizTalk, middleware, all these ease the passage of bits and bytes from one system to another, but none solves the underlying problem, which is only partly a technical one.

systems integration

The issue is particularly acute for those of us who sell ‘best-of-breed’ solutions rather than ‘fully integrated’ solutions. A ‘fully integrated’ solution is one where all the functions you need for your business, from accounting, to manufacturing, to distribution, to HR, can be found in a single large piece of commercial software. Whatever integrations are necessary are already there in the software code itself, and in the database that the entire body of code shares.

A ‘best-of-breed’ solution, by contrast, has its own database and software processes and when integration is required, a separate process must obtain data from one database and pass it to another system.

Why is this difficult?

Part of the problem lies the fact that systems often speak very different languages. I don’t mean that they use different programming languages. They may do that too, but that’s not the point. They speak different languages because they don’t always share a common ‘understanding’ of the ‘events’ which drive them and they don’t usually represent ‘events’ or ‘transactions’ in the same way in their respective databases. They may come close, but the devil lies in the detail. What an ‘invoice’ is, or how it is represented in one system, may be different in another. How an ‘item’ is  represented may differ, and so on. At the heart of ‘fully integrated’ systems lie common concepts of what things are and how they are to be represented by data.

Another problem lies in the detection of change. When systems don’t share a database items that are more or less common to both must be replicated from one database to another (you must usually define which is the ‘master’ or the ‘slave’ in this respect) and the relationship may not be one-to-one. To keep both systems up to date you must detect changes as they occur in one in order to pick them up and copy them to the other. You must do this as rapidly as possible if you are not to cause inconsistencies in the behaviour of both.

And of course ‘fully-integrated’ systems complete their processes in response to an ‘event’ (such as the arrival and booking of a supplier invoice), all at once. Such processes are usually designed so that they entirely succeed or entirely fail. Integrated best-of-breed systems, on the other hand, must do things in sequence. Sometimes it matters if systems get out of step with one other, and sometimes one or more steps may fail.

So when you’re integrating best-of-breed solutions there’s an additional task you must execute from time to time. You’re constantly reconciling your systems to ensure completeness and consistency. This is, in itself, an overhead, but the cost of dealing with failure to reconcile, and the know-how you must possess in order to handle this, add further to your system management costs.

So why would anyone ever choose to buy best-of-breed software systems and go to the trouble of integrating them?

There are several good reasons:

  • ‘Fully integrated’ systems are hugely complex. They must cater for far more variation in implementation. They have many more ‘switches’ that you must know how to control.
  • Their complexity makes them more expensive to buy and to implement.
  • They cause more disturbance to the status quo during implementation because you must throw away everything you already have and start all over again.
  • They’re often not as powerful or clever in some particular areas, despite their complexity, so you’ll sometimes need to choose a best-of-breed solution because it fits your particular needs better. These particular needs may be what makes your company special.

Sometimes, indeed often, the implementation of best-of-breed solutions will be the right choice.

But, what’s the best way of minimising the risks?

Let’s look at that in the next post.

Choosing New Software – Best of Breed or Integrated?

When you’re looking for new software, one of the first things you must decide is whether to look for two or more systems that are best in their particular fields, or one software system that does nearly everything you need.

This choice is between ‘best of breed’ systems:

Best of Breed

or ‘integrated’:

Integrated

The problem with the best-of-breed choice is that you have to do the integration yourself, or commission it. This means developing software to map the data in one system to the data in the other, scheduling the execution of the integration software, ensuring that reference data such as account codes and department codes are synchronised in both systems, and that both systems can be reconciled. It gets even more complicated if there are more than two systems in the mix.

The problem with the integrated choice is that you rarely get everything you want from one system, or, not affordably. And it means that if you only want new software for one particular purpose (e.g. professional services management) then you have to throw away everything else that you have and start again.

It’s not easy to choose.

But if you decide to go the best-or-breed way then Infor’s ION integration framework solves the integration problems by providing technical, logical and procedural management of all aspects of integration.

See this YouTube video to see how we’ve integrated time@work (for Professional Services Management) and Infor SunSystems (for back-office accounting):

http://youtu.be/nS78-f8yU-g