Restricted access

October 25, 2010

Apatar Open Source Data Integration v1.12.13 Release

Filed under: Data Integration, ETL, Open Source — Tags: — Katherine Vasilega @ 12:47 am

New Apatar Open Source Data Integration v1.12.13 has been released recently. It features numerous improvements, bug fixes and new features. Here is a summary of the new release’s highlights.

Apatar users can now copy/paste data components. This functionality is designed to save time on project’s configuration. Now you can copy/paste both single and linked datamap components, selecting any part of the datamap (including connectors, operations, and functions).

Improvements in the connector for E-mail.
There are several improvements in the connector for E-mail: You can now retrieve all or only unread mail from an IMAP mail server; filter E-mails on the server level; set an encrypted connection.

Improvements in database connectors. The other improvements include: displaing views, not only tables in database connectors; the connector for Salesforce CRM is set to prevent from exceeding the number of allowed query characters when updating the tables; specified Long Text fields used at the “Join On” tab of the Join operation into Text fields are transformed automatically, and much more.

You can learn all the details about the New Apatar Open Source Data Integration v1.12.13 release here.

January 5, 2010

Data Integration: Open Source to Become a Mainstream?

Filed under: Data Integration, Open Source — Tags: — Olga Belokurskaya @ 8:09 am

According to multiple predictions and publications, 2010 is going to become quite an interesting year for open source data integration.

Here, as you may remember, Gartner has proclaimed open source solutions “good enough” for data integration (extract, transform, and load, to be exact), and a bit later has mentioned (at last!) open source data integration and BI vendors in its Magic Quadrant, thus admitting that open source solutions can be mature enough to meet their functionality requirements.

Though sometimes there are still talks about the need to have skilled developers at hand, for the sake of support and maintenance, it seems that open source data integration tools move closer to becoming a mainstream, and not just a cheap alternative (with limited possibilities) to proprietary data integration solutions.

Proprietary BI and Data integration vendors seem to admit this fact, as, according to Gartner, some of them have introduced free “starter editions” of their solutions.

All this brings us hopes that times, when open source data integration tools were regarded just offerings for small and mid-sized businesses, are passing, and open source offerings will gain the right to be deployed in large enterprises alongside commercial proprietary BI solutions.

December 18, 2009

On a Couple of Misconceptions About ETL Tools

Filed under: Data Integration, ETL, Open Source — Tags: , — Olga Belokurskaya @ 8:20 am

When deciding to start a data integration process, many companies consider using ETL tools instead of hand-coding. Such a decision is justified by the fact – and many data integration experts agree with it – that hand coding is error prone, takes time and additional resources, etc. However, it’s also wrong to assume that an ETL tool will help to finish data integration project sooner, or will result in some substantial cost savings, according to a TDWI.

Their point is that though ETL tools definitely accelerate the process of data integration at some level, one should not leave aside time that is to be spent on ETL tools evaluation, selection, and implementation.

Another deception is about cost savings. The acquisition cost of ETL tools is quite sufficient, and the annual support cost is often overlooked when a decision is being made on selection and implementation of an ETL tool. Thus, companies have a bit wrong idea about the amount of savings they might have.

The misconceptions described above, are a source of inappropriate expectations, and as a result, wrong assessment of data integration initiative expenses, and at worst, failed data integration initiative.

I think, the situation’s a bit different, when we speak about open source ETL tools. First, there’s no such thing as annual support cost. Huge developer and user communities make it possible to receive support from other users, without paying for it. Then, license costs of open source ETL solutions are really low, which allows to redirect the released budget where there would be a demand for additional finance. So, here I see a real possibility to reduce the cost of data integration with the help of ETL tools.

What I agree with, is that selection process will take time, as well as deployment (including user training), though open source solutions are typically easier to deploy, compared to proprietary ETL tools. Companies should take time for proper evaluation of ETL tools, either open source or proprietary; and I do agree that the decision should be taken based on whether an ETL tool fits this peculiar company’s business needs best and is capable to provide a company with help in achieving their goals.

December 9, 2009

Open Source ETL and BI Tools Deployments to Grow Says Gartner

Filed under: Data Integration, ETL, Open Source — Tags: , — Olga Belokurskaya @ 10:25 am

In one of their latest press releases, Gartner made some interesting predictions regarding open source ETL and other kinds of BI tools. Though it’s early to put open source data integration platforms on the same level with complex proprietary solutions, the deployment of open source tools shows solid growth.

Gartner has pointed out the fact that open source ETL and BI tools are more frequently used by mid-sized businesses, governmental and public sectors. And many ISVs use open source BI solutions as the additional functionality to their own applications.

What is even more interesting: large vendors, proprietary commercial tools providers seriously care about finding the ways to address the challenges from open source data integration and BI offerings, as the latter become their lower-cost competitors.

Assuming this, I think that open source data integration, ETL and other kinds of business intelligence solutions have earned confidence from the business side, and are expected to become even more widely used in the nearest future.

December 7, 2009

Choosing ETL That Fits Your Business Requirements II: Consider Open Source

Filed under: Data Integration, ETL, Open Source — Tags: , — Olga Belokurskaya @ 1:49 am

I suppose this posting to be a kind of a continuation of the previous one. Here I’m again about ETL and data integration solutions selection, but I’d like to concentrate on open source ETL. I won’t make any discovery if, again, repeat that today’s open source solutions are good enough for ETL operations, and  data integration and BI experts are expecting them to develop into solutions for master data management.

But today, open source ETL provides alternative to proprietary solutions which are usually costly and supposed to be used for more complex data integration processes, apart from mere ETL. However, for mid-sized and small businesses that, as a rule, have smaller budgets and smaller open source ETL solutions are a means to address their data integration needs.

But I was about business requirements, or rather how open source ETL tools may address company’s business requirements for data. Here I see several ways:

First, if a company by chance has a couple of their own developers, they could make necessary customization to company’s ETL, thanks to the availability of the code.

Then, as a rule, open source solutions are supported by developers’ communities, some of which are really powerful. So, the community behind the open source ETL that a company uses may help with needed functionality or customization of existing ones to meet company’s business requirements.

And don’t forget about the vendor itself. A company may address directly to the vendor of their open source ETL and require additional functionalities that meet company’s peculiar needs.

And, as a rule, any of the actions described above will cost less and the result will take less time to deliver than in case with proprietary data integration tools.

Well, though the posting sounds so bright, there still may be issues with open source, such as vendors that stop supporting their solutions, etc. However with the communities behind, and thanks to the openness of the code, the chances to overcome those issues seem to me higher than in case with proprietary solutions.

November 16, 2009

Open Source Data Integration and Quality Tools: Being Reasonable

Filed under: Data Integration, Data Quality, ETL, Open Source — Tags: — Olga Belokurskaya @ 2:40 am
While open source technologies gain more popularity as relatively cheap options compared to commercial data management tools, it should be clear that some of the open source tools are more mature, than others, an thus, take care and special attention to what open source may offer for data management, what your needs are, and whether the offerings answer all your needs.

Thus, as most of open source BI software that is now on the market may adequately address the needs of not just SMBs, but quite large companies, many analysts regard them as mature tools.

Almost the same may be said about open source data integration, as there are several mature enough offerings for ETL operations. Moreover, the steps are done toward other forms of data integration.

But, as for data quality tools, open source offerings are still too weak to go as independent option, and not as an addition to proprietary software. Today, such tools may serve as relatively inexpensive means of data profiling.

So, all the above was about to remind of being reasonable and analyzing your needs. While open source may offer mature solutions, not all open source tools are equal, as claims

November 6, 2009

Open Source ETL the Way to Address SMBs Data Integration Challenges.

Filed under: Data Integration, Data Migration, ETL, Open Source — Olga Belokurskaya @ 5:55 pm

The times when data integration from multiple sources was mostly the case for big companies and enterprises have passed. Today more and more companies in small business have lots of various applications and utilize multiple databases, facing sometimes integration challenges.

The thing is, small companies often can’t afford having IT resources for data integration projects. Neither can they afford hiring somebody outside team, for this is quite expensive, plus, there are security questions that always rise when it comes to having somebody from outside work with company’s data.

However, modern ETL and data integration offerings help address small comanies’ data integration challenges providing solutions that cover the lack of IT expertise. Multiple vendors have introduced an alternative for expensive enterprise software, including open source technologies.

Open source data integration tools are the solutions for SMBs to consider. They may not be the answer for full master data management, but cope with classic extract, transform and load (ETL) functions successfully. Providing the possibility to run different operations through graphic UIs, they help avoid costly and complex hand-coding. Thus helping to integrate and migrate data from multiple sources that a company uses.

October 26, 2009

Reasons to Use Open Source ETL

Filed under: Data Integration, ETL, Open Source — Olga Belokurskaya @ 8:00 am

According to a recent survey by Third Nature, open source BI tools, like extract, transform, and load (ETL) solutions, for example, have keeps maturing. Moreover, it becomes more accessible for end-users who have almost no technical background, thanks to enhanced user interfaces, allowing ETL operations performed without hand coding.

There are several more reasons of such an interest to open source data integration tools.

Open source is viewed as a cost-cutting model. The interest in open source tools adoption from this point of view is obvious.

What came as a surprise to me is that open source solutions are preferred to proprietary tools in terms of simplicity. Not having as many functions as proprietary software, open source tools seem to provide “just enough” functionality for data integration initiatives. While there is a tendency for proprietary software vendors to overload their tools with lots of functions which are never used, users seem to need “basic software that works.”

October 16, 2009

Data Integration: ETL vs. Hand-coding

Filed under: Data Integration, ETL, Open Source — Olga Belokurskaya @ 1:49 am

I’d like again to touch upon the use hand-coding and ETL tools for data integration. There seem to be some misunderstanding about what is better and more productive way for data integration. The thing is, that many companies keep hand-coding which is, according to Rick Sherman, an outdated method. Here is why:

Hand-coding is complex, as the amount of data increases and tasks become more complex. Many pages of SQL code for each data source, multiple scripts on duty of gathering data from different sources are hard to keep up-to-date. Moreover, they become more and more expensive in the long term perspective, while the productivity decreases.

ETL tools, while may seem costly on the initial level, use most of common processes, have many possibilities for transformation, and pre-built options to meet different levels of data integration tasks. So, in the long term they turn more productive and cost-cutting, as programmers don’t waste their time and you budget.

Some more about the cost. The times of extremely expensive ETL tools have passed. Today various offerings are available for different budgets and needs. Moreover, there’s a range of open source data integration tools which are, according to experts from Gartner, are really good choice for standard ETL tasks.

October 13, 2009

Open Source Data Integration Competes with Commercial BI Tools

Filed under: Data Integration, Open Source — Olga Belokurskaya @ 1:37 pm

Recent surveys on the adoption of open source BI have shown an increased interest in the kind of tools especially for reporting and data integration initiatives. The respondents mostly admitted active use or the intention to start using open source data integration tools, as well as other open source BI tools. What drives this trend?

Low initial investment in purchase and implementation is a primary reason, as many organizations are being disturbed by the cost-cutting ideas.

Another reason is that open source is no longer a stranger in the world of BI, but a provider of mature competitive tools with a high level of functionality and support, offering such benefits as flexible implementation, access to source code, and ease of integration.

As for support, the community standing behind open source tools turns beneficial solution, for example, for small companies lacking IT staff. Moreover, community contributes codes, new functionalities and modules for the solutions they stand behind.

However, open source tools can’t be named ideal, as there are sometimes challenges users may face with configuration or UIs, and the level of monitoring and management features are not at the same level with commercial BI tools.

« Older Posts