Collaboration and SOA - more than just buzzwords

I tried to answer a question yesterday that I got sent to me via e-mail from an employee at the client company and who is located in Portugal . To be able to answer the question, I had to request information from two persons in my team, one of them being located in Switzerland and one being located in another city in Sweden. In a few minutes I had answered the question and received an e-mail from the employee in Portugal in which he thanked me for the answer. Thanks to the information in my answer, he could continue with his tasks at hand. As I mentioned something about this conversation to my client sitting almost next to me, he said "Isn't it fantastic? That we can sit here and communicate with people all over the world within a few seconds?"

I had to admit that it is quite fantastic. His question got me to reflect on how surreal my current assignment must have seemed just 20 years ago; I am planning and coordinating activities which involve hundreds of people (editors) all over the world, people who are located in countries such as Japan, Sweden, USA, Russia, China, Portugal, and France. I have direct contact with well over fifty of these people, primarily via e-mail or phone. We meet once a year on a conference and then never travel to meet face-to-face.

The project team that I am managing and which is responsible for rolling out new IT solutions to the markets (which we must help all the editors need to learn, prepare and launch) consists of a handful of people. We are partly co-located, but some of the team members work from other locations. We mostly communicate via phone or e-mail and when there is a need for us to meet in a more structured way, we use a web conferencing tool (WebEx from Cisco). We basically never need to travel to get things done. Often, it is just not a viable option. In addition, the corporate policy says that travelling should be avoided for environmental and cost reasons whenever possible.

Although we have much more to do to become more efficient in communicating and collaborating with each other, it is interesting that our biggest headache is not about communicating or collaborating within the project team, with the editors or with all stakeholders in different organizations. No, our biggest headache is the one we get from the constant battle we have to fight with the IT legacy. The problem with the IT legacy is, as usual, that it was not originally designed for the current requirements and that is almost possible to change. The short version is that it is a very complex and lengthy process to get things out to the market (or even at all). Add to that that the inflexible IT legacy adds a lot of extra friction between the ordering organization and the organization supplying it with IT solutions.

I personally both understand and stress the importance of separating the concepts of SOA and web services, one being an architectural style and the other being a technology that can be used for implementing a SOA. Still, I must admit to that web services are excellent for demonstrating key SOA concepts as well as for demonstrating the potential benefits than an organization can get out of SOA. SOA and web services go hand in hand. Even though I now work on the business side and not on the IT side, with people coming from a marketing and business administration background instead of an IT background (I have a combination of both), there is now a common understanding between the ordering organization (“the business”) and the supplying organization ("the IT department") that SOA is the way to go. SOA is no longer just some mysterious or over-hyped buzzword. Why? Because we have all seen a SOA in action.

For a small but essential part of the current solution, core software functionalities and content are now provided to consuming applications via web services. On top of the web services layer, different user experience applications can be designed and developed. Fast. And it can be done by external parties. The latter is virtually impossible today for other parts of the IT platform, even though the task is mostly about presenting existing things in a new way. They will have to navigate in a very complex IT environment and sometimes make changes all the way back to the legacy systems – which of course cannot be changed within the time-frame of the project, so you end up having to scope out functionality and content which would be valuable to end-users. To sum up, new solutions can be delivered much faster since core functionalities and content can be easily reused and thereby shortening development time AND since external capacity can be used to deliver them.

I am absolutely convinced that the keys to empowering enterprises today is to better support communication and collaboration between people and to increase the agility of their IT systems. Improving communication and collaboration is to me a low-hanging fruit since the tools and technologies are out there, although it is a big challenge since it has more to do with people and changing their existing attitudes and behaviors than it has to do with technology. Increasing the agility of the IT systems will in most cases be a costly and potentially very lengthy process, but it is an inevitable investment enterprises need to make if they are to compete on a global and rapidly changing market place.

A real world case of semantic dissonance

My previous post stated that semantic dissonance is the real challenge of content integration and I will try to illustrate this in this post by using a real world case.

Semantic dissonance between two content sources is a typical silo effect, something that happens when two or more information systems that have more or less overlapping content have been developed in isolation from each other, typically to support specific functions within an enterprise. Semantic (as well as structural) dissonance can be expected when information systems developed in two organizations that are to be integrated after a merger or an acquisition. But it is also a common scenario within a single organization.

A couple of years ago, I was working with preparing a new version of an e-commerce site for a global corporation. The site had been in a status quo condition for a couple of years and was now to be redesigned and equipped with a new user experience with more interactive features and richer content. The site was fed with product information such as product structures, descriptions, images, prices and stock quotes from a couple of back-end system, one of which provided the basic product information. The integration with the back-end systems had always been a problem child and not worked properly since it was first designed and implemented several years earlier. The back-end systems where custom developed and the e-commerce platform used for the web site was a standard product.

When we dug a bit deeper into the design and implementation of the current site and how it used the product catalog that was part of the e-commerce platform, it showed that there was semantic dissonance between the product catalog and the back-end system that provided master product information. To put it short, the definition of a product in the e-commerce platform did not match the definition of a product in the back-end systems. Despite the dissonance, the e-commerce site was fully functioning, but mostly due to a number of workarounds. What didn’t work was to administer the products and related content in the administration tool that came with the e-commerce platform. Nor was it possible to use the features that came out–of-the-box with the e-commerce platform – we would have to build them from scratch. Interestingly, the first version of the e-commerce site was designed and implemented by the same company that provided the e-commerce platform.

This insight put us on a long journey where we had to change not only the public e-commerce front-end and how we used the product catalog, but also the integration solution to the back-end systems. We had to convince the business people that this was absolutely necessary to be able to develop the e-commerce sites in the direction they wanted.

The tool that helped us succeed was an information model that helped us distance ourselves from the data models and identify and resolve the dissonance. We (re)defined the term "product" and identified the matching entities in each of the two systems. If looking only at the implementation, it now seemed as we had mapped two different entities with each other. But what differed were only the terms, not the concepts behind them. The information model clarified that the two different terms actually meant the same thing and thus guided the redesign of the e-commerce site and the integration solution.

The trick is how we created the information model. We did not bang the drums and announce to each and everyone that we would now need to redefine the term product and some related terms. That kind of approach would certainly have met resistance because everybody is naturally protecting “their” definitions. To suggest re-defining a term such as “product” in a successful business that was entirely build on their products would probably been political suicide. Instead, we started off with discussing the existing and somewhat inconsistent and confusing product definitions. We did not attack the flaws. We simply said that we needed to understand these terms and their definitions better. So, we modeled and discussed them together in a series of workshops. And behind the scenes, we created the information model piece by piece. It might seem cunning, and it was.

The Real Challenge of Content Integration

Many integration problems cannot be solved by designing computer algorithms. For structural dissonance it is possible, but not if there is some kind of semantic dissonance between (what appears to be) the same content in two different content sources. The term might be the same, but the meaning of it might differ. Dissonance typically occurs:

• When translating real world observations or abstract concepts to information (creating the message)


• When encoding the information into digital content such as text and images (creating the content)


• When transferring content between one source and another where information models do not match (integrating content)

Experience tells that semantic dissonance is common in most enterprises (a silo effect). Still, many choose to put their trust and hopes into integration software, believing that IT alone will solve their integration problems. Reality is of course that any semantic dissonances need to be resolved first, before content sources are integrated technology-wise.

Much more is to be said about this subject, and I will return with discussions and real-world examples on how information modeling can help enterprises overcome semantic dissonance.

Why You Need a Concept Model for Integration

"A human being is part of a whole, called by us the 'Universe,' a part limited in time and space. He experiences himself, his thoughts and feelings, as something separated from the rest - a kind of optical delusion of his consciousness. This delusion is a kind of prison for us, restricting us to our personal desires and to affection for a few persons nearest us. Our task must be to free ourselves from this prison by widening our circles of compassion to embrace all living creatures and the whole of nature in its beauty." - Albert Einstein

An Enterprise Content Architecture (ECA) essentially consists of content (elements) and metadata (attributes and relations). By applying the proper types of metadata to describe and organize the content within an enterprise, the ECA makes sure that it can be managed and delivered as needed. So far so good. But, the main challenge when trying to establish an ECA and integrate content from various content sources is to overcome two great barriers related to the metadata that surrounds the content:

1. The first barrier has to do with the structure of the metadata. If the metadata in one content source is different in structure from the corresponding metadata in another content source, we must make sure that they get the same structure.
2. The second (and even greater) barrier has to do with the semantics – the meaning – of the metadata. If the metadata in one content source does not mean the same as the corresponding metadata in another content source, then we have a problem that we need to resolve. If we are not aware of sematic differences in metadata between two content sources that are to be integrated, then we might get in serious trouble.

To develop an ECA is obviously a challenging - but not impossible - task given that most enterprises have lots of content sources where both the structure and semantics of the metadata have been defined in isolation from how they have been defined in other content sources. This phenomenon iscommonly refered to as the development of “content silos” or “content islands” and it is a result of IT systems being developed to support only specific functions, parts of processes or – at best – entire processes within an enterprise. That is, without an enterprise perspective where all processes and IT systems are viewed as being part of a whole.

To be able to merge different content architectures, there is really no other way than to go back and first (re-)define the basic concepts used in the enterprise. Only when you have established a (agreed upon) concept model on enterprise level you can start to map the content models in different content sources with eachother. Here is a basic approach to establish the enterprise concept model:

1. Make sure you have commitment from top management and then create a team with represenatives from all “content islands” (that is, departments and/or processes)
2. Make an inventory of existing content and which content that is needed but does not exist
3. Identify which content is valuable to the enterprise and therefore should be classified (and managed) as assets
4. Work out an enterprise concept model in modeling workshops

The enterprice concept model can then guide the creation of enterprise metadata standards, enterprise taxonomy development and (structural and semantic) harmonization of metadata in different content sources thoughout the enterprise.

The Role of an Enterprise Content Architecture

The role of an Enterprise Content Architecture (ECA) is to structure, describe, organize and harmonize content resources within an enterprise so that they can be managed and delivered as content products to end users according to business needs and requirements.

One of the main drivers for establishing an ECA is to reduce the costs for producing and managing content. Simply put, an ECA will bring valuable content resources to the surface so that they can be accessed and found. The reverse scenario is that you don’t find the content resources you are looking for and need to re-produce them. Furthermore, if a content product needs to be delivered in different ways – such as via different channels that require different format and structure for the content product - the ECA makes sure that the content resources from which the content product is built can be reused.

What is even more important than reducing costs is that the ECA provides the foundation that enables users (humans aswell as machines) to exchange and share information and knowledge. Is does so by semantically integrating content resources of different formats, structure and types which are otherwise living on their own islands (or kept in silos) somewhere in the enterprise. This is also where the ECA meets the Enterprise Information Architecture (EIA).



While the ECA is primarily focused on supporting the effiency of enterprise content management processes, the EIA is primarily focused on supporting the information needs within the enterprise - to provide the right information at the right time to the right user. Is does so by defining, organizing and describing content products in ways that it supports how different users in different usage contexts look for information and how they want / need the information delivered to them. It goes without saying that need to have both an ECA and an EIA and that they need to harmonize while still being allowed to be different.

An Enterprise Content Architecture semantically organizes content resources that may be of different granularity and be more or less structured. The architecture – relationships between content following certain rules – is created with the use of metadata, such as taxonomies. The ECA also addresses how to structure, describe and store content resources for optimal production, management and delivery of content products to content workers and end users in the business.

In an upcoming post I will look at what kind of questions you need to address when defining and designing an Enterprise Content Architecture.

Insights about Content Management challenges

"Culling Content Management" by Alan Pelz-Sharpe:

"It's long been a gripe of mine that ECM systems were designed to reduce the amount of content you needed to manage to essentials. Yet instead they often just manage everything - trash along with diamonds...//...It's really not that hard to reduce the volumes of content you manage dramatically - a simple content audit can clear out 70-80% without in anyway impacting your RM policies or frankly even being noticable to the end users. Most everything sitting there anyway is a duplication, is redundant or should never have been there in the first place."

"There once was a firm in Nantucket..." by Bob Larrivee at AIIM:

"If your information management and gathering effort is called into question, you may be asked to prove that you have policies and procedures in place that are followed by your employees, using a consistent structure and taxonomy for storing and managing information that is also tracked for purposes of auditing and reporting."

"Strategies for Improving Enterprise Search - Beyond the Out-of-the-Box Experience" by John Ferrara:

"It’s common for enterprise website developers to implement search engines with out-of-the-box functionality, point it at their content repositories, and then just leave it at that. Search is becoming something of a neglected orphan, in part because packaged search products are relatively easy to implement, and then even more easily forgotten...//...Quality search results only come about through applied effort, requiring in particular the skills of an information architect. And IAs must be ready to go well beyond their traditional front-end role, digging into the functional backend and source data of the search engine."

ECM Illustrated - Data and Content Management will blend together

Enterprise users need accurate and complete information in a timely manner to do their jobs. The information they need is often located in different content sources and with different degree of structure. To satisfy the information needs of the employees, an enterprise does not only need to integrate content in different content sources to satisfy information needs, but also integrate structured content (data) with unstructured content.

The world of Data Management has until recently been separated from the world of Content Management, with suboptimization as a natural result. The separation of Data Management and Content Management has its roots in that structured content has traditionally been stored in databases and that unstructured content has been contained in documents and files stored in file systems.

Below is a simple way to illustrate the original positions of Data Management and Content Management and how they will eventually blend together.




From a user perspective, it does not make much sense to treat structured and unstructured content as two separate things. What matters to the user is that he/she can find and access all relevant content and that it gives just enough context to efficiently extract all the information that is needed from the content.

From a business perspective, it does not either make much sense to treat structured and unstructured content as two separate things, even though structured and unstructured content needs to be managed in more or less different ways. The primary concern of the business is to fulfill its goals and to do so it needs to enable the employees to carry out the activities needed to fulfill these goals. One important aspect of this is to supply the employees with just the resources they need, such as content resources from which they can extract the information they need. To treat Data Management and Content Management as two separate things would be as dividing a football (soccer) team into a defensive team and an attacking team and let them play independently of each other. It would be an interesting game to watch, but a sure loss for the team(s).

Enterprise Information Management Components

Information Management is a concept that is relatively well known. Recently the term Enterprise Information Management (EIM) has become popular. EIM initiatives often support a wider range of services and information, are guided by a strategy and driven as a program across business unit borders.

Many people feel it is a daunting task to set off an EIM initiative. How to kick-start and scope a strategy or program? Below is a list of core components to consider with some basic definitions (back-end enterprise applications and other information sources are not included):

Interaction oriented components

• Enterprise Portals: Delivery of digital content in multiple channels
• Self-Service Applications: Automated business transactions initiated by users
• Business Intelligence: Gather and analyse data for business decisions

Information oriented components

• Enterprise Content Management: Production of content for any media
• Information Quality and Integration: Correct and combine information from applications
• Master Data Management: Manage and consolidate shared data

Integration oriented components

• Business Process Management: Model and manage business processes
• Service-Oriented Architecture: Making IT easier to reuse and integrate
• Enterprise Application Integration: Transport and transform messages

Discussing business needs, issues, stakeholders, projects, providers etc in relation to the components above has proven itself a good way to get many EIM efforts going.

Define Before You Integrate

Many organizations believe they have a clear and exact picture of what they mean when they talk about concepts such as product, service and customer. But if you ask around how people in the organization define "product" you will probably get almost as many definitions as the number of persons you ask. Why? Because the seamingly obvious often obscures misunderstandings and inconsistencies. Not many persons dare to question if, for example, a global Telecom company actually has a clear definition of what a product or service means to them. So the concept is left to be defined by anyone who needs it to be defined, without establishing a common definition that is communicated to everyone within the organization.

Many people would get offended if you start asking questions like this. So, is it a stupid question? Possibly. And possibly not. If they can give you a clear definition and show that everybody uses this definition, then it is fine. If not, you have a (political) problem to deal with.

Semantic integration is what integration is about first and foremost, with technical integration coming in on second place. You need first to ensure that "customer" in one business system means the same thing as "customer" in another business system if you are to integrate them, not just make sure that you map attributes in the correct way. And to be able to ensure that, you need a common definition on enterprise level of the customer concept (which might not be exactly the same as in any of the systems to be integrated).

Virtual Content Repositories as Content Integration Approach

In essence, content integration is about providing users within an enterprise with a single point of access to all content they need in their daily work.

There are basically two approaches to content integration – consolidation or federation (point to point integration is basically a swearword today). The consolidation approach, to consolidate all content sources into one single enterprise content repository, must however seem like a utopian dream to most large enterprises. Instead, a common way to provide unified access to heterogeneous content in disperse content sources is to implement an enterprise portal solution with content integration taking place at the presentation level. But this integration approach is in my eyes not "real" content integration since it does not offer the possibility to describe, access and search the content in a unified way. Instead, using a federation approach with a virtual content repository could provide these possibilities “behind the scenes” of the portal solution, with the action taking place in the middleware. The promise of virtual content repositories is just to do that - to provide unified access to disparate content sources within an enterprise without any point-to-point integrations or repository consolidation. Especially the possibility to map metadata between different content sources is essential for content integration. Just as content, metadata is usually stored and managed in isolated islands. As Bruce Silver writes “the user needs to be presented with a unified list of attributes independent of the attribute structure of the underlying systems.”

The major benefits with a virtual content repository approach would be that it is relatively cheap and fast compared to consolidation, and that it will still integrate content so that it can be described, accessed and searched in a unified way. In addition "…virtual repositories can simplify the task of compliance by virtue of containing a single set of business processes applicable to all content in all repositories…//...virtual repositories mean organizations can stop debating whether to go with a single or multiple data stores, and instead concentrate on the critical factors that make for a good repository of any size" (R Dukart)

Major ECM / ECI vendors such as Oracle, IBM, EMC and BEA seem to believe in virtual content repositories for the federation approach, with content being federated to a single virtual repository from any existing content source via a standardized API. Obviously, the key for virtual content repositories to succeed is the use of standardised API:s to access the repository and underlying content sources. JSR 170, the Content Repository API for Java Technology specification developed by Day Software, was the first adopted content repository API standard. The goal with the standard was to “produce a content repository API that provides an implementation independent way to access content bi-directionally on a granular level.” (Day Software). Hence, repositories supporting this standard can be accessed in the same way and the repositories are not tied to any one application. The latest version of the JSR 170 standard, JSR 283, was released in October 2005 by Day Software which leads the specification (and also formed a strategic technology partnership with Oracle in November 2006).

Although still being in an early adoption phase, I believe that virtual content repositories have a future as a content integration approach. Maybe not for "deeper" integration of data in relational databases, but certainly for integrating content such as Office documents, web pages, graphics and e-mails.