Restricted access

March 12, 2010

What Should Data Migration Plan Comprise?

Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 2:22 am

In my previous posting, I wrote about the importance of planning to avoid data migration project failure. So today, I’d like to have some words on what data migration plan should provide for. I mentioned pre-migration procedures and process. A good data migration project starts from planning the necessary pre-migration procedures and then gets to planning the process itself.

Why pre-migration stage? The data can’t be migrated from an old system to a new one just as is, because the old problems will be migrated to the new system as well, thus making data migration useless. To take the most of the new system, a company should ensure the data migrated there can bring value to the business and can be utilized by the business users. Thus, before being migrated:

  • The business value of the data should be analyzed to define what data to be migrated
  • Data cleansing (elimination of duplicate records, etc.) should be performed to ensure the quality of the data.
  • If needed, data transformation should also be performed, to ensure that data formats qualify the new system’s requirements.

Well-elaborated process is the key to data migration project’s success.

  • Data migration project requires creating and implementing migration policies to define the order of the process and a responsible person for each stage of the migration. When the order of the process is set, it’s easier to prevent the troubles, such as server or system crash due to the excessive amount of data migrated at once, etc.
  • Testing is an important stage. One should test each portion of the migrated data to ensure that it’s accurate and in the right format. Without proper testing, the whole data migration project may fail. It’s not a good decision to migrate tons of data only to find out that it’s not in the expected format or the new system can’t read it, and thus the migrated records are useless.
  • In order to ensure future success of data migration project, the process of migration—each stage—should be carefully documented.

So, to conclude: ensure you know what to migrate, provide the quality, systematize the process, test, again test, and document it. This may seem rather time consuming, however, in the reality, when all the procedures and stages are planned, you get more clear picture about time and budget data migration process will require.

September 1, 2009

Growing Demand for Data Integration

Filed under: Data Cleansing, Data Integration, ETL — Olga Belokurskaya @ 11:00 am

Data integration software market shows no sign of slowing. In February, when IT sphere was all in doubts about its future, Gartner predicted the 17% annual growth for data integration software market over the next four years.

So, what makes the data integration professionals expecting high demand for their services?

Well, now it is obvious that data is not just some kind of information to be picked up and stored for the sake of legality or compliance, but that it holds great value to a business. Provided the relations between IT business become closer and closer, there is no surprise that the data integration sector is on the rise.

The understanding that data is the great asset for an organization has also driven up the demand for integration. And it is not simply speaking about providing the basis for sales opportunity and business efficiency, but about data being the foundation for business performance management.

Data integration nowadays is far beyond simple extraction, transformation and loading (ETL). The demand for experts and software capable to cope with such data integration challenges as cleansing of data and improving the data content, mapping of data elements to some standard or common value, transformation of data that does not meet common expected rules, and coping with the situations when those rules fail, is growing as data projects become larger, and ETL implementation is regarded as a prelude to the implementation of corporate-wide, standard business intelligence software.

June 1, 2009

Avoiding 4 Potential BI Cost Factors

Filed under: Data Cleansing, Data Integration — Tags: — Olga Belokurskaya @ 5:46 am

Business intelligence is considered to be the helping hand for organizations wishing to do more with less. However, David Hatch, an Aberdeen Group analyst, has outlined four potential cost factors likely to arise in a BI initiative if an organization isn’t paying attention. According to Hatch, overall cost of ownership is not about the costs of purchasing the software, there are also indirect and hidden factors which affect the real BI costs. Very often the resources the company needs to acquire to properly implement, deploy, support, and maintain a BI solution are far more greater than it was assumed in the beginning.

Here are those factors, as Hatch describes them:

  1. Year-after-year budget increases: The typical best-in-class company sees a drop in year-after-year BI budgetary costs. Average and laggard companies, however, can witness increases in BI expenses that range from 2 percent to 9 percent.
  2. Cost per user: Best-in-class companies lower per-user costs by 4.3 percent whereas average performers and laggards often see increases ranging from 1 percent to 7 percent.
  3. Time to complete projects: Best-in-class achievers complete BI projects, on average, within 14 days. Average performers take nearly three times as long (approximately 39 days) to complete a project, and the typical laggard company takes more than 12 times as long (177 days).
  4. Modifications to BI software: Altering a BI program takes less than a day for best-in-class companies; three days for average performers; and up to eight days for laggard organizations.

Then, how a company can avoid additional costs of ownership and achieve higher returns from BI initiatives? Aberdeen suggests that investments in the following areas will maximize results:

  • Data integration and cleansing: “Companies are finding it difficult to bring data together from multiple, disparate sources,” Hatch says. Investing in tools for data management can be of help in this regard. Best-in-class companies are twice as likely as their counterparts are to institute data integration and cleansing capabilities.
  • End-user requirements: “You really have to stop and think about why…so many companies have deployed tools that so many aren’t able to use,” Hatch says. Companies must understand that end-users — especially nontechnical, non-data-guru types — may need different approaches. Hatch advises companies to focus on end-user needs before deploying a solution.
  • Training: Top performers are 37 percent more likely to invest in extensive user training on BI solutions and 40 percent are more likely to have formed formal user committees to encourage adoption. Additionally, best-in-class companies are twice as likely as laggards and average performers are to sign up for vendor-provided services.
  • Operational BI: Successful users of BI use the technology on an everyday basis rather than merely getting a summarized spreadsheet version of performance and high-level trends. Hatch says that operational BI seems to be gaining traction as companies look to make comparisons over shorter time spans rather than just examine large-scale trends.

May 26, 2009

A Checklist for Data Migration Project. Phase 4: Solution Design

Filed under: Data Cleansing, Data Migration, Data Quality — Olga Belokurskaya @ 12:37 am

At this stage several important questions should be asked and answered about specifications and requirements. By the end of this phase you should have a thorough specification of how the source and target objects will be mapped to pass the results to a developer for implementation in a data migration tool. You also should have a firm interface design, data quality specifications, the firm idea of what technology will be required in the production environment, and, finally, service level agreements.

Here are the details:

  • Have you created a detailed mapping design specification?
    Note that there is no immediate progress into build following landscape analysis. It is far more cost-effective to map out the migration using specifications as opposed to coding which can prove expensive and more complex to re-design if issues are discovered.
  • Have you created an interface design specification?
    A firm design is needed for any interface designs that are required to extract the data from your legacy systems or to load the data into the target systems. For example, some migrations require change data capture functionality so this needs to be designed and prototyped during this phase.
  • Have you created a data quality management specification?
    This will define how you plan to manage the various data quality issues discovered during the landscape analysis phase. These may fall into certain categories such as:

    •     Ignore
    •     Cleanse in source
    •     Cleanse in staging process
    •     Cleanse in-flight using coding logic
    •     Cleanse on target
  • Have you defined your production hardware requirements?
    The volumetrics and interface throughput performance should be known so you should be able to specify the appropriate equipment, RAID configurations, operating system etc.
  • Have you agreed the service level agreements for the migration?
    At this phase it is advisable to agree with the business sponsors what your migration will deliver, by when and to what quality.
    Quality, cost and time are variables that need to be agreed upon prior to the build phase so ensure that your sponsors are aware of the design limitations of the migration and exactly what that will mean to the business services they plan to launch on the target platform.

May 25, 2009

Data Quality Initiatives: Start Small

Filed under: Data Cleansing, Data Quality — Tags: — Olga Belokurskaya @ 8:24 am

Why enterprise-wide data quality initiative often turns into a total disappointment? According to Forrester’s report “A Truism For Trusted Data: Think Big, Start Small,” it happens because of managers’ ambitions to implement an enterprise-wide system for trusted data all at once.

However, experts from Forrester recommend thinking global, but starting small. In other words, they advice consider a bottom-up approach that defines quantitative and qualitative ROI for only those few select functional organizations that can best articulate and measure the business impact poor quality data has on processes.

Here is a short overview of the steps, Forrester recommends, to effectively implement data quality initiatives:

1. First of all, those responsible should start from defining what they mean by “data quality” and “trusted data”. As Jill Dyché once mentioned, it’s time to understand the following seldom-understood truth: That there are different levels of “acceptability” for data. And, according to her, the key is to understand company’s business requirements and then drill them down to data requirements. That will tell conclusively what good enough for the company really is.

Forrester defines “data quality”  as “data used by business stakeholders to support their processes, decisions, or regulatory requirements with no reservations as to the data’s relevance, accuracy, integrity, and other previously agreed upon definitions of quality.”

Forrester reminds that data quality must come directly from business stakeholders, for they are those who understand business requirements of the company, and thus may set the standards for company’s data quality.

Then, Forrester insists on building a business case that starts small.

“Scoping and prioritization based on the business processes within the organization that are most critically affected by poor data quality is the key to defining the business case that will get your trusted data initiative off the ground,” say experts.

3. As word of the value of the data quality project gets around, other organizations, such as those responsible for order management and fulfillment, may also want to implement data quality improvements within their environments.

“Eventually, the tide will turn and these business stakeholders will sign up and support an enterprise-class solution to solving their data quality problems, but for that to happen, value must be demonstrated,” Forrester concludes.

April 13, 2009

Data Quality Steps For Successful MDM Program

Filed under: Data Cleansing, Data Quality — Tags: — Olga Belokurskaya @ 4:57 am

It’s surely no secret that data quality management and MDM are two key factors of enterprise information management. They are interrelated for without DQM, MDM is simply a pile of the data storage as well as DQM cannot bring ROI to the organization without MDM. Actually data quality management plays a role of a building block of an MDM hub as quality and accurate data is a key to the success of an MDM program.

In-depth analysis of the quality and health of data is a prerequisite of the MDM program. Here are data quality management steps suggested at, which are needed to support an agile MDM program:

  1. Identify and qualify the master data and its sources. The definition of master data may be different for different business units. The first step involves identifying and qualifying master data for each business unit in the organization
  2. Identify and define the global and local data elements. More than one system may store/generate the same master information. Additionally, there could also be a global version as well as local versions of the master data. Perform detailed analysis to understand the commonalities and differences between local, global and global-local attributes of data elements.
  3. Identify the data elements that require data cleansing and correction. At this stage, the data elements supporting the MDM hub that require data cleansing and correction have to be identified. Communication with the stakeholders is necessary so that as part of the MDM initiative, data quality will be injected into these selected data elements on an organization-wide basis.
  4. Perform data discovery and analysis. Data collected from source applications needs to be analyzed to understand the sufficiency, accuracy, consistency and redundancy issues associated with data sets. Analyze source data from both business and technical perspectives.
  5. Define the strategy for initial and incremental data quality management. A well-defined strategy should be in place to support initial and incremental data cleansing for the MDM hub. Asynchronous data cleansing using the batch processes can be adopted for initial data cleansing. Industry-standard ETL and DQM commercial off-the-shelf tools should be used for initial data cleansing. The incremental data cleansing will be supported using synchronous/real-time data cleansing.
  6. Monitor and manage the data quality of the MDM hub. Continuous data vigilance is required to maintain up-to-date and quality data in an MDM hub. Data quality needs to be analyzed on a periodic basis to identify the trends associated with the data and its impact over the organization MDM program.

In fact, data quality management is the foundation for an effective and successful MDM implementation. A well defined strategy improves the success probability of an MDM program. Organization should embark a data discovery and analysis phase to understand the health, quality and origin of the master data.

March 18, 2009

Quick Tips on CRM Data Migration Planning

Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 8:26 am

In my previous post i reviewd some points that could help to increase the quality of your CRM database. Now I’d like to touch upon quick tips on what to do if you plan your CRM data migration.

Data migration from one CRM system to another can be quite irksome. Although adding new records to the new configured and ready to use CMR is pretty plain, it’s quite troublesome to shift your previous data into new CRM. Often before the data in your existing format is ready to upload into the new system, it requires a big amount of formatting, enrichment and cleansing. It’s “an inevitable evil” that comes with the migration process.

What you should keep in mind while migrating your CRM data:

Make sure you have an exact back up of all your previous data and the new CRM so that you could roll back to where you were if anything goes wrong.
Check which additional data fields are compulsory in the new CRM and identify them with the fields you have in your current CRM.

Add any additional data items that are missing, remove those that are not required and make sure you have complete records which are ready to be migrated to the new CRM.

As soon as the data moved to the new system, categorize and label it. Do it systematically to avoid the mess and have the retrieval easier.

The whole process, of course, requires a lot of effort and quite dull work, but if it done well, it will be worth every spent minute.

March 17, 2009

Keeping Your CRM Data Quality at Its Best

Filed under: Data Cleansing, Data Quality — Tags: , , , — Olga Belokurskaya @ 4:59 am

What is one of any organizations most valuable assets? CRM data is probably one of them. Companies are likely to protect and secure their CRM data, but what about its quality? Data quality management is very often one of the most neglected areas of CRM management and one of the major pain areas for administrators and managers.

So trying to learn more about the problem, I came across some practices that could help maintain and enhance the value of the data.

  • Do not ignore bad data until it starts affecting your work. Keep an eye on your data and monitor any changes in its quality.
  • It’s a good practice to manage, normalize, format, qualify and filter out your leads outside your CRM and then have it uploaded so that what is not valuable or quality data does not get added.
  • Periodical data append is important although it involves a lot of manual effort and may seem time consuming.
  • Duplication is possibly one of the most common problems and creates redundancy as well as inaccurate reports. So it has to be kept in check
  • The same thing may be said about expired data that simply junks your CRM. The more regularly you check for expired data, the healthier your CRM is.
  • And, finally, if data cleansing is what helps you maintain your database quality then data enrichment is what will help you enhance your data quality and make it more valuable to the end users.

And here we go with the conclusion: good data management practices, constant cleansing and enrichment process – that’s when your CRM data really becomes an asset.

December 3, 2008

Making an Asset of Your Data

Filed under: Data Cleansing, Data Quality, Database Integration — Alena Semeshko @ 4:26 am

Just found a piece from Robert L. Weiner Consulting on the database management. To brush up on your database management strategy, check if you have any of the typical mistakes listed below.

  • Lack of specified data entry policies and procedures, hence no one knowing and applying them.
  • Using Excel as if it were a database (JUST a spreadsheet, okay?)
  • Having no one in charge of data entry/data quality training.
  • Forgetting all about backups, or running them way too seldom.
  • Allowing staff to copy sensitive information onto portable devices and take it home.
  • Insufficient password security measures.
  • Error management? Lack of error management strategies, careless data handling by staff.
  • Keeping your user access rights and security options a mess.

Not about you? Then congratulations, your database management efforts don’t need a fix.

October 20, 2008

The Dos and Don’ts of Data Integration

Filed under: Data Cleansing, Data Integration, Data Migration, Data Quality, Data Warehousing, EAI, ETL — Alena Semeshko @ 2:25 am

Don’t waste time and resources on creating what’s already there.
Extracting and normalizing customer data from multiple sources is the biggest challenge with client data management, according to the Aberdeen Group. OK, true, a lot of companies consider linking and migrating customer information between databases, files and applications a sticky, if not risky, process to deal with. Gartner says corporate developers spend approximately 65 percent of their effort building bridges between applications. That much! No wonder they risk losing lots and lots of data, not even mentioning the time and efforts this may involve. Why spend time on creating what’s already there?

Find an integration provider that suits you. There are plenty of vendors. Of course, there isn’t a universal integrator that would suit everyone, as each tries to cover a a certain area and solve a particular problem. So, you just need to spend a bit of time looking for the right vendor.

Don’t let expenses frighten you.
In today’s enterprises, most data integration projects never get built. Why? Because most companies consider the ROI (Return on Investment) on the small projects simply too low to justify bringing in expensive middleware. Yeah, so you have your customer data in two sources and want to integrate (or synchronize). But then you think “Hey, it costs too much, I might as well leave everything as it is. It worked up till now, it’ll work just as well in the future.” Then after a while you find yourself lost between the systems, the data they contain, trying to figure which information is more up-to-date and accurate? Guess what? You’re losing again.

If ROI is an issue, consider open source software. With open source data integration tools you could have your pie and eat it too. Open source can offer a cost-effective visual data integration solutions to the companies that previously found proprietary data integration, ETL, and EAI tools expensive and complicated.

Not having to pay license fees for BI and data integration software should make companies previously scared of insufficient ROI return to the data integration market.

« Older Posts