Restricted access

September 14, 2010

Data Structure Mismatch during Data Migration Process

Filed under: Data Integration, Data Migration — Katherine Vasilega @ 4:11 am

It’s no secret that one can face serious challenges during the data migration process. Today, I’m going to tell you about one of the most common issues that may complicate the process of data migration or data integration, as the solution may require certain efforts from the side of your technical staff. I’m talking about the mismatch in the ways the data is structured in various systems, such as CRMs, ERP, DBs, and others.

It is often the case that the source system contains different data in one table (e.g., product name, price, description) that has to be migrated to different tables in the target. On the contrary, the source may have different tables (customer name and phone number) for the data that has to be aggregated in one table. Such mismatches are also common for the field names, data formats, and table connections.

For example, the popular CRM systems, such as CRM and SugarCRM, store customer’s name and email in different tables. If you strive for efficient customer management, it’s a good idea to keep this information in one place – so you can view and use your customers’ names and their emails at once. In this case, data migration from CRM and SugarCRM to the target is going to be really time-consuming, unless you connect the different data using an ETL tool. ETL tools provide fast and easy data migration of different types, aggregating unconnected fields into one, or separating one field into several. Data migration with ETL improves your customer relationship management and internal operations, which helps make quality business decisions.

However, a company that is about to start data migration from one system to another for the sake of better customer management should always keep in mind the possible mismatches in how date is structured in these systems. So when it comes to the choice of an ETL tool, ensure that the tool provides for these mismatches and can cope with them. Otherwise, data migration may take much more time and resources than it was initially planned.

April 20, 2010

Understanding Data Integration in the Cloud Context

Filed under: Data Integration, Data Migration — Tags: , — Olga Belokurskaya @ 2:29 am

Today, cloud providers enable both small companies and enterprises appreciate cost-savings, scalability, and ongoing server support, moving their applications or parts of them to the cloud. However, data integration still remains among the challenges cloud providers and their customers face. According to a number of cloud experts (David Linthicum among others), this occurs due to the fact that data integration in the context of the cloud has not yet been clearly understood and elaborated.

  • Sadly, but data integration seems to be an afterthought for many cloud providers; they do not consider the need companies have for synchronizing data in the cloud with the on-premises sources.
  • Another issue is that there are still no common standards for data integration between different clouds. Companies have different business goals and they may use the services from different cloud providers. So, today, we speak about not just cloud-to-on-premises integration, but cloud-to-cloud integration, as well. Since there are a number of different cloud platforms available today, providers are expected to consider cross-platform integration as soon as possible.

So, in the context of the cloud, data integration means the possibility to integrate and manage data across all on-premises and cloud-based systems a company utilizes. Cloud provider should consider these options and give its customers such a possibility.

March 12, 2010

What Should Data Migration Plan Comprise?

Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 2:22 am

In my previous posting, I wrote about the importance of planning to avoid data migration project failure. So today, I’d like to have some words on what data migration plan should provide for. I mentioned pre-migration procedures and process. A good data migration project starts from planning the necessary pre-migration procedures and then gets to planning the process itself.

Why pre-migration stage? The data can’t be migrated from an old system to a new one just as is, because the old problems will be migrated to the new system as well, thus making data migration useless. To take the most of the new system, a company should ensure the data migrated there can bring value to the business and can be utilized by the business users. Thus, before being migrated:

  • The business value of the data should be analyzed to define what data to be migrated
  • Data cleansing (elimination of duplicate records, etc.) should be performed to ensure the quality of the data.
  • If needed, data transformation should also be performed, to ensure that data formats qualify the new system’s requirements.

Well-elaborated process is the key to data migration project’s success.

  • Data migration project requires creating and implementing migration policies to define the order of the process and a responsible person for each stage of the migration. When the order of the process is set, it’s easier to prevent the troubles, such as server or system crash due to the excessive amount of data migrated at once, etc.
  • Testing is an important stage. One should test each portion of the migrated data to ensure that it’s accurate and in the right format. Without proper testing, the whole data migration project may fail. It’s not a good decision to migrate tons of data only to find out that it’s not in the expected format or the new system can’t read it, and thus the migrated records are useless.
  • In order to ensure future success of data migration project, the process of migration—each stage—should be carefully documented.

So, to conclude: ensure you know what to migrate, provide the quality, systematize the process, test, again test, and document it. This may seem rather time consuming, however, in the reality, when all the procedures and stages are planned, you get more clear picture about time and budget data migration process will require.

March 9, 2010

The Role of Planning in Data Migration

Filed under: Data Migration, Data Quality, Uncategorized — Tags: , — Olga Belokurskaya @ 3:00 am

Data migration has never been an easy process, and though there are a variety of tools available today, the process remains complex, and the rate of errors is high. Data integration may fail due to hardware or system failures, but those are so-called unforeseen situations. The most common reason for migration project failure is lack of proper planning.

In the result of rushing into migration without careful planning of time and resources needed, data migration projects experience schedule delays and require additional expenses, so budgets get overrun. That’s because multiple issues occur during the process of data migration, including copy process failures, issues with data formats match in the source and the target, server crashes due to excessive amounts of data migrated at once, etc. Coping with these issues requires time and money, so data migration process may stick.

Proper migration planning should include a set of pre-migration procedures and well-elaborated migration program to help address data migration complexities, hit deadlines, and avoid unpredicted additional costs. I’ll touch on this in my next posting.

March 3, 2010

Data Integration Is Not About Tools, It’s About Strategy

Filed under: Data Integration, Data Migration, ETL — Tags: , — Olga Belokurskaya @ 4:13 am

Today, organizations face increasing data integration challenges. The amounts of data grow progressively, demanding for new levels of data protection, and making data migration even more complex.

At the same time, business demands access to a real-time information and quality data to make right business decisions. There are plenty of technologies that are able to address data integration challenges, though some of them get old-fashioned, some continue to mature, etc. Thus according to Forrester Research, MDM and data quality services continue to mature, while ETL (extract, transform, and load) and data replication “have reached the Equilibrium phase,” and some technologies are moving to a decline.

But that doesn’t mean that a technology or tools which are in the top of the list today, may become a successful solution for data integration challenges. Lots of organizations regard data integration as mostly a technological process, not taking into account how it impacts organization’s long-term plans and the success (or failure) of business. However, successful data integration is mostly about strategy. And when a strategy is defined, the choice of tools get’s much easier, and there’s no risk that a chosen technology won’t cope with the task.

February 25, 2010

What May Complicate Data Migration?

Filed under: Data Integration, Data Migration — Olga Belokurskaya @ 1:38 am

Data migration is a complex process, and it differs greatly from other IT projects. Good sound approaches should be introduced when data migration strategy is being planned. Success of data migration depends on many things, each detail is important. But there are certain conditions that affect the complexity of data migration process, and thus demand for even more attention.

One of such challenging conditions is moving from a current vendor to a new one, which may mean another type of applications and systems, different data formats, etc. This may, probably, demand the use of data migration tools different from those, a company had utilized.

Another condition that complicates data migration is moving from physical environment to virtualized one, such as the Cloud. Though the end result of the shift is going to be great, a much greater effort is needed to overcome difficulties, connected with the process (lack of interoperability between different cloud and physical platforms, security provisions, access to the data, etc.)

So, as new possibilities for storing, accessing, and working with data appear, promising significant decrease in expenses and resources, organizations will adopt them. However, certain things and conditions that may complicate the process of data migration should be taken into account, and companies should provide for them.

February 18, 2010

Data Migration: Challenges of Moving Data to a CRM

Filed under: Data Integration, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 4:36 am

CRM is a great solution to effectively manage company’s customer data. However, to ensure efficiency, get the most out of CRM system, and avoid CRM failure, special attention should by paid to the data that is being migrated to a CRM, and how it is being migrated. There are challenges that may affect the process of data migration:

  • Migration from heterogeneous sources. Data migration from various sources to a CRM is a challenge, so it should be approached with care and strategy. Migrating the data just as is leads to a CRM failure, as you simply take siloed data and move it to another system (the data still remains siloed). So, before data migration takes place, there is a work that should be done to ensure the quality of the data that is targeted to a CRM.
  • Data mapping. To migrate data, one should map the source and the target. That’s clear. The problem is, that in different sources data fields possessing equal information, may be named differently (for example, “Username” and “Account”), or vice versa, equally named fields may stand for different values (“Name” may stand for a first name solely, or for the full name, including first name and surname). That’s why it’s very important to map fields in a data source with appropriate fields in a CRM application.

It’s highly probable that before data migration is started, data will need to be transformed, in order to make it appropriate for a new CRM system. Surely, it should be cleaned from duplications, incomplete records, outdated records, etc. Only when data is prepared, clean and in the appropriate format it may be migrated to a CRM.

January 12, 2010

Data Migration: Ensure Quality When Moving Data to the Cloud

Filed under: Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 5:57 am

When migrating data to the cloud, ensuring data quality is essential.  Data can’t be taken to the cloud as is, so before starting data migration, provisions should be made for data quality.

It’s wrong to start data migration, until data is checked to be accurate, complete, duplications are found and cleared up, etc. Otherwise, data issues will be taken to the cloud, which will make it inconvenient to work with the data.

One more thing to be taken into account before beginning data migration process is a provider’s possibility to provide fresh, real-time data, and give constant access to the data.

Normally, companies have best practices for data quality. Cloud providers also have tools for data management, so when those tools and company’s best practices are united, it makes data management more flexible, and thus a company will have possibility to control data quality when data is migrated to the cloud.

December 24, 2009

Data Migration: Challenges of Moving to the Cloud

Filed under: Data Migration — Tags: , — Olga Belokurskaya @ 8:35 am

Okay, the cloud seems to become a real trend, so now for many enterprises there’s no question whether to adopt the cloud, but how to do it better.  Cloud’s major idea, that attracts many organizations, is the possibility to move their applications out of internal data centers into the cloud, and let cloud providers take care about maintaining the applications. However (and it’s been mentioned not once), application migration and data migration to external clouds are not simple processes.

Not to become overenthusiastic and have a clear view on moving to the cloud, it’s good to know the challenges and questions that data migration raises.

Clouds possess different architectures, and thus require technical personnel to upgrade their skills to fit cloud requirements in terms of implementation and operation. This is the most likely-to-face data migration challenge when moving applications to the cloud.

Data security is also among cloud data migration challenges. In fact, moving data into some third-party provider’s servers, you’ll know about it that it is somewhere in the cloud. This somewhere may be anywhere; there were lots of talks this year about clouds lacking transparency. However, this doesn’t mean that data migration into the cloud is a bad idea. This only means, that what applications (and what data) to move to the cloud should be carefully defined.

As for lowering cost, yes, moving to the cloud is one of the solutions to cut company’s expenses.  But that won’t come immediately. Initially, data migration will demand investment, for either data migration tool will be necessary to perform on-premise-cloud data migration, or the need to pay for data migration service. What matters, is that there won’t be free data migration anywhere. Depending on what tool you will choose, the cost may be higher or lower.  Yes, I’m talking about open source solutions that are available today as widely, as proprietary tools. However, when making a decision about what to choose, you shouldn’t be driven by merely cost cutting ideas, but follow your business goals, and choose the tool providing best opportunities to achieve those goals.

So, moving to the cloud has become a trend, and it’s fair, for the benefits provided of the cloud are real. However, though it seems simple just to move enterprise applications and data to the cloud, data migration must be properly planed and possible challenges analyzed before making the first move.

December 23, 2009

Cloud Data Integration Issues: Why Large Enterprises are Reluctant to Adopt Cloud Services

Filed under: Data Integration, Data Migration — Olga Belokurskaya @ 8:37 am

First of all, are they reluctant? According to an article at ebizQ I’ve stumbled upon, large enterprises “typically don’t deploy latest versions of anything, anytime soon.” This is more than fair when we speak about cloud computing. Small and mid-sized companies adopt cloud services much faster, notwithstanding the fact that cloud adopters may face security and data integration issues.

So, why data integration with the cloud is considered to be a complex problem?  There are different cloud platforms available today. Consequently, each cloud platform has its own API, having its own functionality and quality of services, which differ greatly from other cloud platforms’ APIs. So, here’s one of the reasons of the complexity of data integration between enterprise and cloud sources: additional efforts will inevitably be needed to provide for this difference.

Getting back to SMBs and large enterprises, SMBs are more flexible in adoption of evolving technologies. That’s because most SMBs are do not possess, or simply can’t afford IT departments responsible for data integration and other company’s IT needs. So cloud services offer real way out for such companies. As for data integration, small companies have relatively small amounts of data to be integrated between on-premise applications and applications residing in the cloud (that’s surely, compared to large enterprises).

Large enterprises have different requirements to data integration: their expectations and operational rhythm differ greatly from those of SMBs and mid-sized organizations. So there’s no surprise large enterprises are not hurrying into the cloud, waiting until all issues, including data integration, compliance and security, to be solved.

« Older PostsNewer Posts »