Posted in Architecture

In part 1 of this series I discussed traditional reasons people have for buying software, which turned out mostly to be based on perceived or real issues with building software. In this post, which contains part 2 and 3, I discuss issues with buying software.

Part 2: Traditional issues with buying

Motivating the decision to buy software by pointing out the problems with building software can be effective, but it implies an important assumption, namely that there are no issues with buying software or that these issues are smaller than the issues with building software. It should be pretty obvious that there are no approaches without issues, especially not in IT. So, what are the problems that organisations have encountered when buying software products? And how big are these problems?

Software packages deliver features that are not needed

Companies that write and sell software packages face an interesting challenge. Usually each of the organisations they are selling to have somewhat different needs and processes. The overlap can be sizeable but even in the best case there will be features that are only required by a subset of the potential customers. At the same time it is normally too costly and complex for the software vendor to produce individual packages for each of their customers. Their business model is based on the fact that the software can be written once and sold many times. Responding to this challenge most vendors choose to implement all features that all of the potential customers may need.

A variation of this theme is software that is split into modules. Large ERP systems for example have modules such as payroll and logistics. In these cases, however, the same challenge repeats at the module level, not every company's payroll requirements are the same. To make matters worse, the customer now has to choose, install and maintain several pieces of software.

In addition to responding to known needs of their customers the software vendors, especially in competitive markets, exert force on each other. They must implement ever more features to differentiate themselves from the competition. At this point features required across most of the customer base are considered tickets to the game, features that just have to be in the product to be considered. The differentiating features are often aimed at smaller segments of the audience or more speculative in nature, which normally leads to a very large feature set, and to users that only need a fraction of that feature set. An extreme case is Microsoft Word. A blog post on Microsoft's developer site reports that out of hundreds of commands in Word "five commands account for around 32% of the total command use in Word 2003"[3]. There are varying estimates for business and enterprise software but often the number of features used by a specific customer or user is estimated to be below 10% of the overall feature set.

When all the features needed by the users at a given customer are available one could consider this a success. The users have their requirements met and the software vendor has a model that works for them. Unfortunately, there is a problem: All those features that are present in the software but that are not needed by a given user add overhead for the customer and their users. First and foremost, the features are visible and at best may just take up space in menus, toolbars, etc but at worst may confuse the users or make use of required features more complicated. This often translates to lost productivity and increased training effort. The additional features are also likely to result in a higher resource usage, such as memory and storage space as well as network utilisation, which translates into higher capital expenditure for hardware and increased operational costs.

Software packages require customisation

One way to address the issues outlined in the previous section is to customise the software so that unneeded features are not visible or not even installed. This way, users are not burdened with these features and the features do not use additional resources. However, any customisation of packaged software brings with it the problems discussed in the context of developing software. Even if it is to a smaller degree, customisation brings risk in that unforeseen issues can cause delays or increase cost, they take time to carry out and test, thus increasing the time to introduce the package, and customisations require competencies that are not core to the business that is acquiring the package.

In most cases customisations are needed beyond removing unneeded features. Business, even those that operate in the same market, providing the same services to their customers, usually employ somewhat different internal processes and practices, and obviously the software package needs to support these. Some packages basically store data and make it accessible at a later stage. In such cases customisation is limited to adjusting the structure of the data and the screens to access it, which is normally relatively painless. More and more often, though, software packages actively support the workflow of an organisation and in such cases customisations quickly become more complex.

Adjusting workflows and processes in a software package exposes the customer of the software package to several equally unappealing options: They can change the actual source code of the package, thus effectively forking the codebase. Apart from the upgrade issues discussed further down, this brings buying software even closer to developing it in the first place, with all the issues discussed before.

Another option is to choose a software package that allows customisation through rules engines and workflow engines. The hope is that following a formal business process modelling exercise, producing a clear description of the business processes in a standard notation such as BPMN, the workflow engine can simply use that model without the need to change any code. In practice, however, it turns out that formally modelling the business processes is actually hard. In fact, many of the problems and risks associated with developing software are not caused by the actual programming but by getting the requirements right, and that is closely tied to understanding the business processes at a detailed level. Furthermore, capturing complex processes in a graphical notation adds its own set of problems. Even with sophisticated tooling navigating and manipulating large diagrams can be more cumbersome that working with a textual representation such as code. Features that facilitate collaboration between several modellers, such as version control involving merges and conflict resolution, are nowhere near as mature in workflow tools than in those tools available to manage source code.

A middle ground between source code and graphical notations are textual configurations and domain specific languages. In the end, they address some of the concerns, but still do not resolve the fundamental issue that any customisation adds risk and complexity.

Now the key question is: How much risk and complexity do customisations add? Unsurprisingly, there is no simple answer. Sometimes customisations are small and negligible in the context of the introduction of the software package. In many cases, though, the customisation can be as big or bigger than entire software development projects. Colleagues of mine have reported instances of customisation code alone exceeding 100,000 lines and I have seen a system in the financial services industry that comprised more than a million lines of XML code describing how to handle so called corporate actions. To be fair, this was not based on a software package for corporate actions, if such a thing even exists, but on a more general processing engine package.

An interesting approach to avoid the issue entirely was pointed out by my colleague Martin Fowler. He suggests that "you should buy a package and adjust your business process to fit the way the package works"[4]. As mentioned in Martin's article and earlier in this one, an organisation should not use software packages in areas that are a strategic differentiator for them. In consequence, if packages are used only in areas that are not strategic to them then it does become possible to change the process to match the software, thus avoiding the need for customisation. This is especially true for smaller and medium sized businesses, who in fact may benefit from improved processes enshrined in the software package.

Some companies, of course, choose to use packages in areas that are strategic to them and even then there are examples of companies eschewing customisation. In one case an entire mobile phone network was set up with packages from a large software vendor and the network's CIO "insisted in the set up [...] that there would be no customisation" and "he believes he bought in best practice methods [...] and teams around the business should, and have, aligned their methods with the applications."[5]

Software packages bring their own roadmap

Yet another dimension to be considered is the evolution of the software package. With the exception of cases such as software to handle taxation in a specific year almost all software packages evolve over time as the software vendor responds to feedback from their customers and their evolving needs. New versions may be released as infrequently as once a year or they could come out at short and regular intervals.

Each update brings incremental change and the effort required to carry out the upgrade mounts with each update the customer does not install right away. Therefore the question for most customers is not so much whether to upgrade but when to upgrade. The vendor's product roadmap now needs to be taken into account when planning the internal roadmap and work required to upgrade needs to be scheduled. Depending on how much customisation was required to adapt the package the effort to upgrade can be significant.

One way to reduce the effort is to reduce customisation as described before. I have heard of cases where customers decided against getting access to the source code of a package, even though that was offered for free and would have allowed them to make some necessary customisation more easily. They did so because they knew that by changing the code they would effectively be forking the vendor's codebase and would have to reapply all their changes with each new version. Instead the customers, or more precisely their IT departments, decided to live with workarounds that inconvenienced their users. This is another example of conflicting interests of the IT department and the business leading to a worse outcome for the users. However, in this case it is not just risk but actual effort that the IT department is avoiding, and it can make good business sense to do so.

An extreme solution to the upgrade issues is not to upgrade the software package at all. The customer buys a version of the software that works for them, customises it to fit its needs, and then stays with their customised version indefinitely. Generally, this not an option, though, because updates often include fixes for security problems and with the degree with which software is networked unfixed security problems are not tolerable. If the customer is lucky the package vendor provides security fixes as patches that can be applied independent of upgrades, but even then, at some point the current version of the package will have changed so far from the version the customer is clinging on to, that the patches are no longer directly applicable.

Lastly, over longer periods of time the ideas a package vendor has for its product may diverge from the needs of a given customer. The outcome in this scenario very much depends on the relative size and power of the two companies. If the package vendor is much bigger than the customer then the customer has no choice but to either move into the new direction and adapt to the changes or they have to procure and install a different package and absorb the cost of doing so. If the customer is a very large enterprise and the package vendor is somewhat dependent on them, then the customer can impact the direction into which the product is evolving. They may ruin the chances for the package vendor in the long term, though. In either case, there is significant risk or cost for the customer, something that buying software is meant to reduce in comparison to developing software.

Part 3: A new issue with buying

The issues discussed above are issues that organisations have had to to deal with when buying software for a long time. Now, advances in IT and corresponding changes in the business have introduced a new issues with buying software.

Software packages struggle with service orientation

Traditionally software packages were mostly applications that solved a specific need in the business, such as the stereotypical example of a payroll system or software to calculate risk and premiums for an insurance. The applications were often stand-alone system with little integration between them. Following a growing need for more detailed reporting, and enabled through better database technology, the 1990s saw the widespread adoption of data warehouses. New, tool-supported extract transform load (ETL) processes took data from several operational systems and placed it into a central data warehouse where it could be aggregated and analysed. Despite the ability to view data across the organisation most IT landscapes were still very much application-oriented.

Reporting was only one step as businesses looked for ever tighter integration of their processes and business units. In areas that relied heavily on batch-processing, often run during the night, or on manual re-keying, sometimes mocked as swivel-chair integration, the expectation was that data should be processed straight-through, that is it should flow from one boundary of the IT systems landscape to the other immediately and without manual intervention. This desire led to a widespread move from application-oriented landscapes to service-oriented architectures (SOA). In the most successful cases existing line of business applications were decomposed into a portfolio of services, resulting in tighter integration, more consistency and less duplication. Whether a service-oriented architecture is generally desirable is not the topic of this article. It is clear though that many businesses and enterprises, maybe even a majority of those who rely on IT solutions, are moving towards them. With this change new challenges for software packages arise.

Even without a fully developed SOA a new software package must be integrated into a complex IT landscape, a landscape that may have grown over decades. In such cases configuration and customisation, with their associated effort and risk, are joined by integration work, the effort of which is rarely negligible. Worse, though, integration works usually carries a lot of risk and that risk is hard to mitigate because the integration effort involves other software systems.

Today's large ERP packages, for example, have about 3,000 interface points, usually in the form of web service endpoints and operations. These operations are defined by the package vendor, which means that they are unlikely to match exactly the requirements and standards of the systems they are to be integrated with. In fact, they are often thin shims over internal data structures. All this means that the package does not only have to be configured and customised but that specific integrations must be written. Sometimes the impression is created, usually with reference to standards such as SOAP and WS-*, that systems can simply be plugged together by drawing lines in graphical tools. Unfortunately, though, in the real world this is not normally possible. The standards, even if fully interoperable implementations are found, are too low level and do not cover enough of the semantics of the integration. From personal experience I can also attest that even major implementations of seemingly basic standards such as WS-Security are not always complete or fully interoperable.

As an organisation moves from siloed applications to a service portfolio the integration effort and risk in relation to the overall effort and risk increase. Not only do key benefits of buying packages erode, but because the packages by definition have more rigid integration points than custom developed software, they also make this growing area of effort and risk more difficult and more risky.

There are two solutions to deal with the integration problem. One approach is to procure all packages from a single vendor, an approach that is sometimes called best-of-suite, I guess not for linguistic accuracy but to contrast it with best-of-breed. In a way this is the logical conclusion of the idea to shift risk from the buyer of software to the vendor at the possible expense of features, fit or upfront cost. Interestingly, many of the large suites sold by software vendors are the result of mergers and acquisitions, which means that the integration work remained necessary but it really did move from the buyer to the vendor. The mobile network example cited above is an example of this strategy.

Another approach to mitigate integration risk, when following a best-of-breed strategy or when integrating bespoke services, is to buy the integration itself in the form of software products such as an enterprise service bus (ESB) or even a complete business process integration suite. The theory is that the integration product removes the burden of having to write integration code and makes integration just a configuration task, which is obviously the same idea as buying and configuring software packages. In practice, however, and certainly not for a lack of trying, such integration products usually do not deliver the benefits expected. As discussed above in the context of graphical modelling tools, the problem is that the tool attacks what is easy, namely the mechanics of the integration, but provides little to no support to tackle what is hard, the semantics of the integration. If two fields in two XML documents can be connected easily in a graphical tool then even in the easiest cases somebody has to find out whether account numbers are zero-padded, whether currency symbols, and which, are included with monetary values, whether a boolean field contains zero and one, or yes and no, or true and false, etc. That combined with the additional effort to master the tool usually by far exceed the effort to connect two custom-developed software components.

In part 4 I conclude this series by describing approaches for organisations that want to shift more towards building software.

References

[3] Jensen Harris, "No Distaste for Paste (Why the UI, Part 7)". MSDN Blogs, 7 Apr 2006

[4] Martin Fowler, "Package Customization". 6 July 2011

[5]"Business must align with IT", CIO UK