Restricted access

October 1, 2010

Data Integration: 3 Most Common Mistakes

Filed under: Data Integration, Data Quality, ETL — Tags: , , — Katherine Vasilega @ 4:51 am

Implementing a data integration solution is not an easy task. There are some common mistakes that companies tend to make in data integration. These mistakes result in delayed data integration projects, increased costs, and reduced data quality. Today, I’d like to focus on three most common data integration mistakes that businesses tend to make.

1. Lack of a comprehensive approach

Data integration is not only about gathering requirements, determining what data is needed, creating the target databases, and then moving data. You have to develop a comprehensive data integration approach that will provide for:

• Processing complex data, such as products and customers, in relation to facts, such as business transactions
• Filtering and aggregating data
• Handling data quality
• Capturing changes and avoiding gaps in historical data.

2. Missing data quality requirements

You may think that data quality problems are simply data errors or inconsistencies in the transactional systems that can be easily fixed. The truth is that you have to prevent quality problems at the initial stage of a data integration process. You have to plan how to set data quality requirements, incorporate data quality metrics into your system architecture, monitor those metrics in all your data integration processes, and report on data quality.

3. Using custom coding instead of ETL

While most businesses consider ETL the best practice, there are still a lot of companies that use custom coding to create countless data shadow systems. Keep in mind that custom code makes it difficult to manage and maintain programs, does not offer the centralized storage of programs, limits metadata capabilities, and also has a longer development cycle. Besides, debugging is more difficult with a custom code than with an ETL tool. To add more, an ETL tool usually has a user-friendly interface, provides for centralized storage of programs, and is relatively easy to customize.

Thinking ahead about all these issues before developing and implementing a data integration solution, you are sure to save time, money, and valuable data.

September 23, 2010

Data Integration to Achieve Data Quality

With an ever-increasing amount of data coming from various sources, you are sure to face data quality issues. Should you maintain a huge database of contacts and send notifications, sales offers and other documents to all of them? Isn’t it way too time-consuming and cost-ineffective? Wouldn’t it be smarter to check, which contacts do have a potential of becoming your customers and whether they really exist at all?

One of the ways to tackle the issue is to implement demographics-focused solutions in data integration of your CRM system, Web site membership database, Excel documents, and other data sources. For example, data integration with CDYNE Demographics Web service will allow you to receive relevant information about contacts from any U.S. postal address before you launch an advertising/marketing campaign. With a help of the appropriate ETL software, you can integrate this Web service into your customer database to determine contacts’ age, nationality, income or other characteristics, such as type of residence, average income, average house value, average number of vehicles for residents in their neighborhood, etc.

You can also integrate your CRM or any other contacts database with StrikeIron Email Verification service. This data integration solution allows instant determining the validity of an email address or domain. You can check all of your contacts and send emails to those that actually have them.

Data integration with demographics-focused solutions and address verification software ensures enhanced data quality, which results in better customer service, effective marketing and advertising, and, eventually, increases your revenue.

March 12, 2010

What Should Data Migration Plan Comprise?

Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 2:22 am

In my previous posting, I wrote about the importance of planning to avoid data migration project failure. So today, I’d like to have some words on what data migration plan should provide for. I mentioned pre-migration procedures and process. A good data migration project starts from planning the necessary pre-migration procedures and then gets to planning the process itself.

Why pre-migration stage? The data can’t be migrated from an old system to a new one just as is, because the old problems will be migrated to the new system as well, thus making data migration useless. To take the most of the new system, a company should ensure the data migrated there can bring value to the business and can be utilized by the business users. Thus, before being migrated:

  • The business value of the data should be analyzed to define what data to be migrated
  • Data cleansing (elimination of duplicate records, etc.) should be performed to ensure the quality of the data.
  • If needed, data transformation should also be performed, to ensure that data formats qualify the new system’s requirements.

Well-elaborated process is the key to data migration project’s success.

  • Data migration project requires creating and implementing migration policies to define the order of the process and a responsible person for each stage of the migration. When the order of the process is set, it’s easier to prevent the troubles, such as server or system crash due to the excessive amount of data migrated at once, etc.
  • Testing is an important stage. One should test each portion of the migrated data to ensure that it’s accurate and in the right format. Without proper testing, the whole data migration project may fail. It’s not a good decision to migrate tons of data only to find out that it’s not in the expected format or the new system can’t read it, and thus the migrated records are useless.
  • In order to ensure future success of data migration project, the process of migration—each stage—should be carefully documented.

So, to conclude: ensure you know what to migrate, provide the quality, systematize the process, test, again test, and document it. This may seem rather time consuming, however, in the reality, when all the procedures and stages are planned, you get more clear picture about time and budget data migration process will require.

March 9, 2010

The Role of Planning in Data Migration

Filed under: Data Migration, Data Quality, Uncategorized — Tags: , — Olga Belokurskaya @ 3:00 am

Data migration has never been an easy process, and though there are a variety of tools available today, the process remains complex, and the rate of errors is high. Data integration may fail due to hardware or system failures, but those are so-called unforeseen situations. The most common reason for migration project failure is lack of proper planning.

In the result of rushing into migration without careful planning of time and resources needed, data migration projects experience schedule delays and require additional expenses, so budgets get overrun. That’s because multiple issues occur during the process of data migration, including copy process failures, issues with data formats match in the source and the target, server crashes due to excessive amounts of data migrated at once, etc. Coping with these issues requires time and money, so data migration process may stick.

Proper migration planning should include a set of pre-migration procedures and well-elaborated migration program to help address data migration complexities, hit deadlines, and avoid unpredicted additional costs. I’ll touch on this in my next posting.

March 5, 2010

Database Integration: On the Importance of Data Quality Standards

It’s a sad fact, but many organizations realize the poor quality of the data in their databases, only when it comes to database integration. Data quality issues are among the common reasons for data integration failure.

This neglecting attitude to data quality lies in the fact that companies often don’t understand how much data quality impacts business processes. Thus, each data source or database a company uses may have its own rules and standards for data quality. The issues, however, evolve as soon as the database integration started in order to get a unified look at, for example, company’s customers’ data.

Those issues may come out of the difference of data fields, for example, or data formats, so the same contact may be represented differently in different databases. Thus when it comes to database integration, it can’t be performed correctly due to those differences, which may lead to data duplication, and many more data quality issues. In fact, in the result of integrating several databases of poor quality, a company gets one big database of poor quality. This means that database integration was in vain, as it failed to achieve its main goal of providing the company with a general view of business data, while the integration expenses were significant.

Unfortunately, data quality technology does not always allow organizations to fix poor data. So, it’s much wiser to implement company-wide standards for data quality to prevent the appearance of data quality issues associated with integration of data from heterogeneous sources, then to perform data cleansing and other data quality procedures afterward.

February 18, 2010

Data Migration: Challenges of Moving Data to a CRM

Filed under: Data Integration, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 4:36 am

CRM is a great solution to effectively manage company’s customer data. However, to ensure efficiency, get the most out of CRM system, and avoid CRM failure, special attention should by paid to the data that is being migrated to a CRM, and how it is being migrated. There are challenges that may affect the process of data migration:

  • Migration from heterogeneous sources. Data migration from various sources to a CRM is a challenge, so it should be approached with care and strategy. Migrating the data just as is leads to a CRM failure, as you simply take siloed data and move it to another system (the data still remains siloed). So, before data migration takes place, there is a work that should be done to ensure the quality of the data that is targeted to a CRM.
  • Data mapping. To migrate data, one should map the source and the target. That’s clear. The problem is, that in different sources data fields possessing equal information, may be named differently (for example, “Username” and “Account”), or vice versa, equally named fields may stand for different values (“Name” may stand for a first name solely, or for the full name, including first name and surname). That’s why it’s very important to map fields in a data source with appropriate fields in a CRM application.

It’s highly probable that before data migration is started, data will need to be transformed, in order to make it appropriate for a new CRM system. Surely, it should be cleaned from duplications, incomplete records, outdated records, etc. Only when data is prepared, clean and in the appropriate format it may be migrated to a CRM.

February 16, 2010

When Developing Systems Architecture Think About Data Integration

Filed under: Data Integration, Data Quality — Tags: , — Olga Belokurskaya @ 6:34 am

“The data, and integration strategies around the data, is something that most figure is there, will be there, and requires very little thinking and planning.” – This is what I’ve read today at ebizQ.

This again supports the idea that data, though being “the biggest companies’ value”, is still often being neglected. And this results in data integration and quality issues, providing inconsistent data and ruining the entire idea of data integration as a way to provide a clear view on enterprise data. The data that is important for business decisions.

In fact, very often when it comes to designing and developing the architecture, all the attention is focused on technical side of the process. Thus data integration strategies become an afterthought making it difficult to meet business requirements. So, the message is that provisions for data integration should be made at the level of the development of enterprise systems architecture.

February 12, 2010

Data Integration and Data Quality Require Consolidation

Filed under: Data Integration, Data Quality — Tags: , — Olga Belokurskaya @ 10:08 pm

Lately, there have been talks about the need for a union between data integration and data quality solutions, as no integration initiative can be called successful without the quality of the data provided.

Today, the functionality that provides data quality becomes a requirement for data integration tools. However, data quality still remains a challenge. According to Gartner, only 25% of organizations will proactively and comprehensively include data quality processes and competencies in their data integration work in 2010. This might mean that a large number of enterprises will have to struggle with incomplete, inconsistent and inaccurate data, as their main focus is on the mechanics of data integration process with no connection to data quality.

Technology providers apply more efforts to combine data integration and data quality processes into one data management process. Steps have already been made toward the consolidation of technology offerings, instead of using distinct data integration and data quality solutions. The benefits of this consolidation are quite obvious – a consistent and complete view of the business data, and as a result, the ability to change business processes and models faster to gain the market.

January 15, 2010

Customer Data Integration Among the Top Priorities for Companies

Filed under: Data Integration, Data Quality — Tags: , — Olga Belokurskaya @ 8:35 am

According to one of the latest researches by Aberdeen Group, issues with customer data may greatly affect organizations’ sales and marketing efforts. Besides, lack of trusted customer data and inability to target customers through customer data proved to be the top two of those issues, according to the respondents’ opinion.

This reveals the fact, that there’s still a lack of customer data integration and data quality at many organizations.

This seems strange to me, especially taking into account that experts have been talking a lot about the importance of customer data integration across the enterprise to give all the responsible a better view on who are their customers, what do they need and buy.

Even in a rather small company there could be several sources containing customer data, and not obviously those sources contain similar records, so customer data integration is the means to piece together the data puzzle.  Probably, mergers the number of which was significant in 2009 prevented numerous organizations from fulfilling their customer data integration initiatives.

Well, the fact is that more than 60 percent of Aberdeen’s survey respondents named customer data integration and other customer data initiatives among their top priorities for 2010.

January 12, 2010

Data Migration: Ensure Quality When Moving Data to the Cloud

Filed under: Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 5:57 am

When migrating data to the cloud, ensuring data quality is essential.  Data can’t be taken to the cloud as is, so before starting data migration, provisions should be made for data quality.

It’s wrong to start data migration, until data is checked to be accurate, complete, duplications are found and cleared up, etc. Otherwise, data issues will be taken to the cloud, which will make it inconvenient to work with the data.

One more thing to be taken into account before beginning data migration process is a provider’s possibility to provide fresh, real-time data, and give constant access to the data.

Normally, companies have best practices for data quality. Cloud providers also have tools for data management, so when those tools and company’s best practices are united, it makes data management more flexible, and thus a company will have possibility to control data quality when data is migrated to the cloud.

« Older PostsNewer Posts »