Gartner has lately named cloud computing among top 10 strategic technologies for 2010. However, that doesn’t mean moving to clouds immediately, experts say, but to consider the ways to approach the cloud, meaning the use of cloud services, private cloud environments deployment, applications development, etc.

Why experts are warning against hurrying into the cloud? One of the reasons is that making certain applications work effectively in the cloud, especially, legacy ones, is quite difficult.

There are several factors (according to an article at Computerworld), and the variety of cloud platforms is one of them. That means that data integration and data migration processes will be different within each platform, as well as costs, support functions, etc.

Then, the means of data integration like APIs provided by the cloud differ from those used in a company’s systems, which may demand inventing some solution to address this integration challenge.

One more essential difficulty factor is that most of legacy apps hardly support the newest technologies cloud platforms are built with.

Thus, while cloud computing named strategic, there are lots of things to be done about data integration, security, compatibility issues, etc. before migrating legacy applications to the clouds. Firstly there should be ways found to address those issues and ensure the apps work efficiently.

According to a recent survey by Third Nature, open source BI tools, like extract, transform, and load (ETL) solutions, for example, have keeps maturing. Moreover, it becomes more accessible for end-users who have almost no technical background, thanks to enhanced user interfaces, allowing ETL operations performed without hand coding.

There are several more reasons of such an interest to open source data integration tools.

Open source is viewed as a cost-cutting model. The interest in open source tools adoption from this point of view is obvious.

What came as a surprise to me is that open source solutions are preferred to proprietary tools in terms of simplicity. Not having as many functions as proprietary software, open source tools seem to provide “just enough” functionality for data integration initiatives. While there is a tendency for proprietary software vendors to overload their tools with lots of functions which are never used, users seem to need “basic software that works.”

Customer data being one of the main assets for organizations is needed to be backed up regularly. Backups prevent important data loss in case something happens.

To have your customer data backed up to a secure remote location, you may want to explore Amazon S3 (Simple Storage Service), which lets you easily store and retrieve virtually any amount of files anytime, anywhere. Amazon S3 deploys the same highly scalable, reliable, fast, and inexpensive data storage infrastructure that Amazon.com uses to run its own global network of websites. Apatar’s Amazon S3 connector brings the power of Amazon S3 to Salesforce.com users who may want to store or back up Salesforce.com customer data and documents.

For instance, if a company’s executive wants to have his or her company’s most significant customer information backed up every day (e.g., extracted into flat files, and saved to Amazon S3), the Apatar tool allows for this data to be backed up and then uploaded to Web storage at a specified time. All you need to do is configure the Amazon S3 connector and enter the frequency and the moments of SalesForce.com data backups in Apatar’s Scheduling module. The ETL engine will do the rest automatically.

To find out more about integrating Salesforce.com with 3-rd party applications, have a look at our white paper on “Five Steps to Integrate SalesForce.com with 3rd-Party Systems and Avoid the Most Common Mistakes.“

It’s been written so much about challenges of data integration initiatives, about all those mistakes and issues that organizations face starting from the preparatory level, so when, at last data integration at an enterprise starts working everybody sighs happily.

However, they miss at list one thing to worry about. It’s performance. What’s wrong with performance, you say. Well, according to David Linthicum, computing and application integration expert, there are two main data integration performance issues:

The first occurs due to the organic data growth. Being a common issue for both real-time and batch data integration model, it results in the system failing to get and deliver the needed data in time, providing undesirable latencies.

The second issue is a result of bad data integration architecture. This happens when core data integration performance requirements have been poorly formulated, or wrongly understood. The mistake leads to wrong technology or approach selection.

What’s so dramatic about it? Well, let’s think. Why someone starts data integration initiative? Let’s say, to have a better look and access to data needed for business. If this data doesn’t appear in target systems when it needed by the business process, the goal of data integration initiative was not achieved.

So, plan data integration architecture properly and keep an eye on performance.

I’d like again to touch upon the use hand-coding and ETL tools for data integration. There seem to be some misunderstanding about what is better and more productive way for data integration. The thing is, that many companies keep hand-coding which is, according to Rick Sherman, an outdated method. Here is why:

Hand-coding is complex, as the amount of data increases and tasks become more complex. Many pages of SQL code for each data source, multiple scripts on duty of gathering data from different sources are hard to keep up-to-date. Moreover, they become more and more expensive in the long term perspective, while the productivity decreases.

ETL tools, while may seem costly on the initial level, use most of common processes, have many possibilities for transformation, and pre-built options to meet different levels of data integration tasks. So, in the long term they turn more productive and cost-cutting, as programmers don’t waste their time and you budget.

Some more about the cost. The times of extremely expensive ETL tools have passed. Today various offerings are available for different budgets and needs. Moreover, there’s a range of open source data integration tools which are, according to experts from Gartner, are really good choice for standard ETL tasks.

To ensure the process of integrating Salesforce.com data with 3rd-party applications and databases to be successful, it’s important to know how to solve typical challenges and avoid the most common mistakes at each of the steps that should be taken to get customer and enterprise information (currently residing in SalesForce.com) integrated, replicated, or migrated to some new Software-as-a-Service package.

The first step includes preparation and planning. It’s essential at this level think over and clarify the goals of the oncoming integration process:

• What data (tables/fields/rows) should be extracted?
• What data (tables/fields/rows) should be considered as targets?
• Do I need to integrate SalesForce.com with one single database or multiple data sources?
• Is it enough to perform a one-time migration, or do I need an ongoing synchronization?
• Do I need to have SalesForce.com data backed up?
• Do I have enough experience to do manual coding, or would the use of visual data integration tools be the best decision?

Mind that having no strategic vision and not enough evaluation criteria are the most common mistakes occurring at the preparation level. So it’s really important to set the goals and objectives properly.

Find out more about successful SalesForce.com integration from our whitepaper on “Five Steps to Integrate SalesForce.com with 3rd-Party Systems and Avoid the Most Common Mistakes.”

Recent surveys on the adoption of open source BI have shown an increased interest in the kind of tools especially for reporting and data integration initiatives. The respondents mostly admitted active use or the intention to start using open source data integration tools, as well as other open source BI tools. What drives this trend?

Low initial investment in purchase and implementation is a primary reason, as many organizations are being disturbed by the cost-cutting ideas.

Another reason is that open source is no longer a stranger in the world of BI, but a provider of mature competitive tools with a high level of functionality and support, offering such benefits as flexible implementation, access to source code, and ease of integration.

As for support, the community standing behind open source tools turns beneficial solution, for example, for small companies lacking IT staff. Moreover, community contributes codes, new functionalities and modules for the solutions they stand behind.

However, open source tools can’t be named ideal, as there are sometimes challenges users may face with configuration or UIs, and the level of monitoring and management features are not at the same level with commercial BI tools.

Recently, I’ve touched upon the topic of data integration with ODBC. Today, I’d like to add some more words. Successful data integration with ODBC sources depends on many things of which not the least is the performance of ODBC applications. There are several factors that affect ODBC performance. Improvements of those factors help make ODBC applications faster which, in turns, help improve and avoid issues in data integration. Here the factors are:

1. Network communication
   Reducing network communication may increases ODBC performance multiple times. Arrays of parameters used instead of Insert statements, for example, reduce the time required to complete the operation.
2. Choosing the way the transactions are handled
   To improve ODBC performance it’s essential to choose the right way transactions are handled. Thus, for example, using manual commits instead of auto-commits gives better control over the work committed.
3. Connection pooling
   When an ODBC application has several users connection pooling is a good way to increase connection efficiency.
4. SQL queries
   Efficiency of SQL queries is an important factor affecting the speed of ODBC performance. If something is wrong with it, issues may occur with data filtering causing the driver to get unnecessary data (sometimes the amount of this data is very big) which slows down application performance. Using well-formed and rightly executed queries improves the performance greatly.

ODBC provides good opportunities in data integration, giving an access to multiple data sources through one application. So keeping the application’s performance high will benefit the process of data integration.

While clouds keep gaining popularity, there are issues occurring from time to time, as more organizations step into clouds. Of those issues, one of the most important is the data issue. More exactly, companies that moved their data to the cloud, sooner or later face the fact, that it should be somehow integrated or synchronized with the data in on-premise enterprise applications.

In fact, the possibility of data integration between on-premise and cloud systems and tools is the thing a company should provide for before turning to clouds. However, this fact is often overlooked, which leads to data integration problems in the future, taking into account the amount of data that is going to increase.

Here, some issues to consider before moving to clouds, provided by David Linthicum, a computing and application integration expert:

• Firewall limitations – there should be the way found to externalize and consume not port-80-compliant data.
• The speed of moving data should be considered in order to customize transformation and routing mechanism to perform properly.
• Provide for maintenance and support for cloud systems.
• And security, which is still a big issue for clouds.

To conclude, there should be planning, provisions for data integration, and technologies thought over properly to address issues if they occur prior to migrate any systems to the cloud.

We all got used to speaking of data integration in terms of extract, transform, and load (ETL) mostly. And that’s pretty fair, for gathering and transforming data from one location and putting it into another location has always been, and still is the major task for data integration.

However, according to Rick Sherman, a data management expert, new trends make data integration step out of mere ETL borders, as technologies and processes evolve helping data integration tools turn data into “comprehensive, consistent, clean and current information.” Many tools support processes like data migration, data profiling and quality, application consolidation, etc. The time when IT departments had to build those processes into data integration, have passed, and tools appeared with all above mentioned functions pre-built.

Thus, enterprise data integration initiatives that once were extremely time-consuming more and more tend to become real-time as business demands more current information.

One more thing, Sherman dwells on is hand-coding being an out-of-date practice. Frankly, why keep using error-prone hand-coding, when there is a wide range of ETL tools available. There is a choice in configuration and price making it possible to find the tool to fit one’s needs. Taking into account open source ETL tools available almost for nothing, the devotion to hand-coding seems pretty strange or, at least unwise.