Data Integration Blog

There’s really no universal criteria for selecting a data quality provider/solution, it’s rather personal for each company. Before looking at different providers the smart thing to do is to decide on the list of priorities to look for in potential solutions. Some give priority to accuracy, others find execution time the most critical element.

Andrew J. Brooks, wrote a post in his blog on datamigrationpro.com discussing his top three must-be’s for a data quality solution. Come to think of it, I wouldn’t argue with his top-3. It’s Engineered - Understood - Trusted. I’ll look at them step-by-step.

1) What he calls engineered, I’d rather call integrated actually.

It’s about making data quality management an integral part of your architectural design principles; it’s about culture change and cannot be solved by buying a tool.

The solution needs to be fully integrated into all the work processes and become a part of the company’s overall strategy and performance.

2) Understood.

Having accurate, complete, relevant meta data, reference data, master data – call it what you will, is one hell of an obstacle that many have thought about, and most have failed at.

So, a more organized approach than many companies have would definitely work better.

3) Business Trust.

Without business trust, no amount of data profile reports will ‘make’ the business use your data for decision making.

Trust, that never really occured to me. Focusing on your data and leaving out networking and building trust-based relations with the business players around you, as a lot of companies tend to do, is definitely the approach that lacks wisdom.

The article “Tension emerges between SOA and mashup camps“makes a very good point about SOA vs. mashup relations.

Mashups make SOA real to business users - nothing wrong with that

…you have SOA practitioners on one hand, calling mashups ungovernable and Web 2.0 camp saying “SOA poisons the mashup well”.

Mashups are simple, convenient and helpful. I don’t see any reason for them not to be popular. If SOA is a complicated concept, how is it not natural for users to  favour a nice and sweet shortcut?

SOA is seen as complex, while mashups seen as an easy shortcut to agility.

“…The approach is not a black and white SOA vs. mashups choice for enterprise integration, but rather, use of mashups for the last mile of integration that may, in many cases, utilize data services, feeds, or other sources that more often than not are exposed as Web or RESTful Services.”

So it’s not the SOA vs. mashups, it’s the SOA crowned with mashups that would be the right solution. With this approach the corporate world will be able to “see and feel and touch service orientation.”

I just came across a superb article that chews data integration out for you the best possible way! For business users that do not write code, it contains a comprehensive description of data integration approaches to keep in mind when selecting an integration provider to work with. I personaly consider manual integration, where you need to do all the work and write code quite limited as compared to “application approach”, where a ready application will do everything for you.

“The applications, which are specialized computer programs, would locate, retrieve and integrate the information for you. During the integration process, the applications must manipulate the data so that the information from one source is compatible with the information from the other source.”

“Most data integration system designers assume that the end goal is to create as little work for the end user as possible, so they tend to focus on applications and data warehousing techniques.” That’s the idea: the easier for the end user (I particularly mean corporate users that cannot afford to lose time on integrating their customer lists and data manually), the better!

News from SugarCRM! The company introduced Sugar Data Center Edition (DCE) – a new product line for SugarCRM partners and enterprise customers, a set of provisioning and monitoring tools enabling service providers and large organizations to deploy and manage multiple instances – distinct versions of SugarCRM – from a centralized management console. Sugar DCE will be delivered in two versions – one for partners and one for customers. Sugar DCE for Partners allows SugarCRM resellers and hosting providers to deliver SugarCRM in a Software-as-a-Service (SaaS) model to their customer base. Sugar DCE for Enterprises allows large companies to manage multiple versions of SugarCRM within their company from a single location.

Is SugarCRM already deployed in your company? Do you keep important customer information in a database or legacy CRM and want to migrate that data to your new SugarCRM account? Ever thought of integrating it, or your SalesForce.com customer list with that in your SugarCRM? Working with SugarCRM on a corporate level becomes more and more attractive.

Savio Rodrigues has posted a brief review of Apatar in his InfoWorld Open Source weblog.

The really cool thing is that Apatar provides a visual designer and mapping tool in order to hide the complexity of ETL and data integration from the typical (business) user.

Yep, that’s the idea. Apatar was primarily created and designed as a business user-oriented tool. On top of visualization that Apatar provides, most of the things the user has to do manually is drag-and-drop icons to the visual integrator. That is, no coding whatsoever. And yet another of Apatar’s corporate user-oriented features is the connectivity itself. Apatar provides connectivity to corporate aplications like SalesForce, SugarCRM, GoldMine, etc. and can be used by pretty much any business user not familiat with development at all.

Software-as-a-Service market only continues to grow, presenting a more and more efficient and reliable business model. According to Gartner, by 2011 25% of new business software will be delivered as SaaS. This SaaS boom makes more and more companies that have never before even considered moving to SaaS-type business model, make decisions in favour of adopting it. Howevever, as always with large and popular projects, the essence and the very milestones get trivialized, you get a wrong picture of it and end up somewhere totally different from what you planned.

William McKnight has a recent entry in his blog uncovering the essence and how-tos of the process of selecting a SaaS vendor.

The 3 hallmarks or selling points of SaaS he names are:

1) no IT involvement,
2) pay-as-you-go with little upfront cost
3) the vendor takes all responsibility for infrastructure and upgrades – those invasive and non-value-added activities.

And some rules of thumb to keep to when selecting a SAAS vendor/solution:

1. Check the value proposition of the application.
2. Ensure the ability to expand the core functionality of the application beyond the provided functionality (scalability).
3. Don’t sabotage IT plans.

For more details check out William’s post.

There’s an insightful article by Mike Karp on ILM (information lifecycle managememnt) and the six steps of implementing a successful and efficient policy on data storage, verification, classification and management. Mike identifies the following steps to follow to ensure your ILM efficiency:
Stage 1. Preliminary
1) Determine whether your company’s data is answerable to regulatory demands.
2) Determine whether your company uses its storage in an optimal manner.

Stage 2. Identifying file type, users accessing the data and key words used.
1) Make a list of regulatory requirements that may apply. Get this from your legal department or compliance office.
2) Define stakeholder needs. You must understand what users need and what they consider to be nonnegotiable.
3) Third, verify the data life cycles. Verify the value change for each life cycle with at least two other sources, a second source within the department that owns the data (if that is politically impossible, raise the issue through management), and someone familiar with the potential legal issues.
4) Define success criteria and get them widely accepted.

Stage 3. Classification (aligning your stakeholders’ business requirements to the IT infrastructure).
0) Identifying the business value of each type of data object, i.e. understanding three things: what kind of data you are dealing with, who will be using it and what its keywords are.
1) Create classification rules.
2) Build retention policies.

When you engage with the vendors, make sure to understand their products’ capabilities in each of the following areas:
* Ability to tag files as compliant for each required regulation.
* Data classification.
* Data deduplication.
* Disaster recovery and business continuity.
* Discovery of compliance-answerable files across Windows, Linux, Unix and any other operating systems you may have.
* Fully automated file migration based on locally set migration policies.
* Integration with backup, recovery and archiving solutions already on-site.
* Searching (both tag-based and other metadata-based).
* Security (access control, identity management and encryption).
* Security (antivirus).
* Set policies to move files to appropriate storage devices (content-addressed storage, WORM tape).
* Finding and tagging outdated, unused and unwanted files for demotion to a lower storage tier.
* Tracking access to and lineage of objects through their life cycle.

Finally, when you know your vendor, you can look for solutions to automate the needed processes and phase-in.

See full article for more details.

Dion Hinchcliffe wrote an article discussing the recent mashups’ popularity boost.

“Penetration of mashups in the enterprise is just beginning as their benefits begin to be understood.”

Mashup Tools and Platforms Circa 2008 (The diagram by Dion Hinchcliffe from the article mentioned above)

Well, I couldn’t agree more. Mashup technologies are the next generation approach to infromation integration and management.

The figure above illustrates the mashup market distribution very well, But! =)  Apatar is placed in the code mashup section for developers, which doesn’t exactly illustrate its real market placement . Apatar is a strictly data mashup tool designed for business users. It can, of course, be used by developers, but with the sole purpose of simplifying their work and avoiding writing code. Apatar positions itself as a No-Coding tool aimed at making mashups easy and available for consumers and power users (not developers). So, Apatar should’ve been somewhere in the upper left side of the diagram. =)

As companies go on struggling to get a positive return on investment from SOA, we hear more and more about SOA prjects failing. However what we don’t see is that software oriented architecture, just like its predecessor, integration, is a complicated concept that has its own challenges and risks.

Among the challenges SOA presents are:
it’s dynamic: the products, standards and requirements keep on changing and maturing, and keeping up with the flow might not be the easiest thing. For companies that have adopted SOA early on this dynamic process is twice as hard to follow.
it’s a twist: adopting SOA requires that global changes are made in your project lifecycle management, as SOA has a very specific infrastructure that needs significant scalability capacities, a fully altered and specific work and design pattern, user training and performance.

These two challenges associated with adopting SOA enterprise-wide can, however, be addressed.
Here are a few of the risk mitigation steps that you could take as suggested by Eric Roch in his recent blog entry on ITtoolbox:

� Examine the current architectures and methodology in use and adjust for SOA - an agile OOA/OOD approach with specific SOA deliverables and patterns
� Establish a repository and governance policies for reusable artifices such as interface specifications (design deliverables), schemas and interface definitions (WSDL)
� Develop SOA reference architecture based on design patterns, tool usage and best practices that defines the SOA logical and physical architecture
� Establish a training plan for staff competency
� Acquire message based testing tools and develop SOA Quality Assurance policies and procedures
� Involve operations support early and deploy monitoring and management tools for the SOA infrastructure
� Develop a SOA strategy and roadmap based on business value, risk, business process effectiveness, and IT assets to be leveraged
� Transition to SOA iteratively adding services based on business value and utility of function building the services library over time

This article by David Linthicum suggests that people seem to be confusing integration with SOA these days. The thing is, integration came first and later on, when SOA came to take its place, a lot of vendors simply started calling their integration tools SOA. David notes, “integration, on its own, is not architecture. Thus, just binding systems together is not architecture, thus not SOA.”

However, if SOA is new and, you can’t argue, the idea behind both of these contepts is fairly similar (instead of building new and duplicating old applications and data, the technology should connect the existing vital parts and build up on them)… doesn’t it sound like it’s just another case of a fresh PR for an old, but improved technology?