Restricted access

December 29, 2009

On Approaches to Data Integration with the Cloud

Filed under: Data Integration — Tags: , , — Olga Belokurskaya @ 2:25 am

The idea of cloud computing has become really popular; the increase of the cloud adoption in 2010 has been predicted by data integration experts. There’s no surprise, as cloud computing is positioned as a way of simplifying technology, so that wider audience could adopt it.

The need to synchronize data from applications that an enterprise has in the cloud with on-premise apps is bringing us to cloud integration (or data integration with the cloud).

Today there are different approaches to data integration with the cloud available, as well as different solutions to perform cloud integration. This variety is mostly comes out of the immaturity of the market, and, I suppose, with the development of the cloud, there will be defined right approaches to data integration. Among the most common approaches is customizing the same data integration tool that an enterprise uses to integrate in-house applications, so it could be used for the cloud applications, as well; moving a data integration tool to the cloud, which helps avoid expenses connected to hardware installation. One more way is on-demand data integration.

So, there is a variety of choices, and to make a decision on what approach to utilize for cloud data integration a company should understand their needs, as well as consider available budget and resources, etc.

September 29, 2009

Getting Ready for Data Migration: Data Quality Issues

Filed under: Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 1:42 am

Careful analysis of the quality of data in your current system should be done prior to migration. Good data can save quite a bit of time (and budget) during migration. In fact, this analysis and replacement set up during the migration will help you cleanse and improve your data. Below are some points to pay attention to.

  • Required fields
    • - There fields that require being filled in for records to be migrated, such as company names for accounts, customer names for contacts, and so on. The main issue here is some of the fields missing information.

      To cope with the issue, think what values should be entered into these empty fields. The key factor here is to make it convenient for you to use the data when it is migrated.

  • Data type transformations
    • - It is important to ensure that values entered in the fields of one system meet the data type requirements of the system where the data is migrated, and there is no conflict during the migration.

      - One more issue is data logical duplicates that are presented in different ways in the field. The thing is that there may be different ways of naming the same value. So you need to replace all the different names with one. If there’s no possibility for automatic replacement in your current system, you’ll need to add these replacements within the data transformation to be built for the project.

    Learn more about the data quality issues that may occur during data migration from the whitepaper on “The Three Most Common Data Integration Problems

    September 21, 2009

    Understanding System Capabilities to Ommit Data Migration Issues

    Filed under: Data Migration — Tags: , — Olga Belokurskaya @ 7:44 am

    As your business grows, requirements for your data storage change. At the same time, the IT market offers new, diverse software to catch up with or even get ahead of your needs. At a certain stage, you face the necessity of switching to a new system. What are the hidden problems related to data migration (DM)? How can you omit them and get the most of your DM initiative?

    Studying the new system’s capabilities is an important step not just in preparing requirements for your data migration/integration project, but in selecting the system in the first place. Here are some things to do to ensure your choice is right, and you understand the system:

    • Make sure the system really does meet your requirements and will be able to satisfy your needs.
    • Make sure the components and functions/capabilities of the system are convenient for you to work with and perform the actions you need to perform with your data.
    • Analyze how the new system communicates with other systems. A good import/export mechanism included in the new system is a useful feature, but it’s not always sufficient. So, it is important to study the system’s API or SOAP capabilities in order to define whether the system you’ve chosen provides a full-scale toolset for data integration or additional alternative integration methods will be needed, such as direct database access, if it is enabled by the system. This issue will definitely make the project more labor-intensive, time-consuming, and, as a result, more costly.

    Learn more about the challenges of data migration in our white paper “The Three Most Common Problems Faced During Data Migration

    May 20, 2009

    A Checklist for Data Migration Project. Phase 2: Project Initiation

    Filed under: Data Migration, Data Quality — Tags: — Olga Belokurskaya @ 1:20 am

    Continuing my previous post, here are the questions to ask oneself at the stage of data migration project initiation:


    • Have you created a stakeholder communication plan and stakeholder register?
      During this phase you need to formalize how each stakeholder will be informed.
    • Have you tweaked and published your project policies?
      Now is the time to get your policies completed and circulated across the team and new recruits. Any policies that define how the business will be involved during the project also need to be circulated and signed off.
    • Have you set up your project collaboration platform?
      This should ideally have been created before project initiation but if it hasn’t now is the time to get it in place.
    • Have you created your standard project documents?
      During this phase you must create your typical project documentation such as risk register, issue register, acceptance criteria, project controls, job descriptions, project progress report, change management report, RACI etc. They do not need to be complete but they do need to be formalized with a process that everyone is aware of.
    • Have you defined and formalized your 3rd Party supplier agreements and requirements?
      Project initiation is a great starting point to determine what additional expertise is required.
    • Have you scheduled your next phase tasks adequately?
      At this phase you should be meticulously planning your next phase activities so ensure that the business and IT communities are aware of the workshops they will be involved in.
    • Have you resolved any security issues and gained approved access to the legacy datasets?
      Get approvals from security representatives (before this phase if possible) and consult with IT on how you will be able to analyze the legacy and source systems without impacting the business.
    • Have you defined the hardware and software requirements for the later phases?
      What machines will the team run on? What software will they need? What licenses will you require at each phase? Model re-engineering tools? Data quality profiling tools? Data cleansing tools? Project management software? Presentation software? Reporting software? Issue tracking software? ETL tools? It can often take weeks to procure this kind of equipment so you ideally need to have done this even before project initiation.

    May 19, 2009

    A Checklist for Data Migration Project. Phase 1: Pre-Migration Planning

    Filed under: Data Migration, Data Quality — Tags: — Olga Belokurskaya @ 3:26 am

    Data migration is a complicated and complex process, so it is need to identify the risks and activities well in advance. A proper planning of the process is an essential part of a data migration project.  A good structured plan may help not to overlook some points, which may prove to be crucial for the whole project, and thus lead to project delay or closure.
    I’ve recently found a very good and detailed set of questions one should consider when embarking on a data migration project by Dylan Jones from Data Migration Pro Journal, and I’d like to share it.

    Here are the items one should check on a pre-migration phase:

    • Have you assessed the viability of your migration with a pre-migration impact assessment?
      It is advisable in a pre-migration level to verify the cost and viability of the migration, including its terms (how long it will take), the choice of technology it will require and provisions for dangers that may lie ahead.
    • Have you based project estimates on guesswork or a more accurate assessment?
      Provide accurate analysis of cost and resource requirements so if you have tight deadlines, a complex migration and limited resources make sure you perform a migration impact assessment asap.
    • Have you made the business and IT communities aware of their involvement?
      It makes sense to inform the relevant data stakeholders and technical teams of their forthcoming commitments before the migration kicks off. There are numerous aspects of the migration that require business sign-off and commitment. Make sure everyone understands and agrees to what their involvement will be.
    • Have you formally agreed the security restrictions for your project?
      Obtain a formal agreement from the relevant security governance teams in advance. Simply putting your head in the sand and hoping you won’t get caught out is unprofessional and highly risky given the recent loss of data in many organizations.
    • Have you identified your key project resources and when they are required?
      The essence is to understand the key migration activities and dependencies then plan to have the right resources available when required.
    • Have you determined the optimal project delivery structure?
      Agile, iterative project planning with highly focused delivery drops is far more effective than a classic waterfall design, so ensure that your overall plan is flexible enough to cope with the likely change events that will occur. In addition, it’s wise to provide for delay occurrence due to contingencies.
    • Do you have a well defined set of job descriptions so each member will understand their roles?
      Have an accurate set of tasks and responsibilities for each member of the team involved in the project. Clear understanding of what your resources need to accomplish will help you be fully prepared for the project initiation phase.
    • Have you created the appropriate training documentation and designed a training plan?
      Data migration projects typically require a lot of additional tools and project support platforms to function smoothly. Ensure that all your training materials and education tools are tested and in place prior to project inception.
    • Do you have a configuration management policy and software in place?
      Data migration projects create a lot of resource materials. Profiling results, data quality issues, mapping specifications, interface specifications… Ensure that you have a well defined and tested configuration management approach in place before project inception.
    • Have you planned for a secure, collaborative working environment to be in place?
      If your project is likely to involve 3rd parties and cross-organizational support it pays to use a dedicated product for managing all the communications, materials, planning and coordination on the project.
    • Have you created an agreed set of data migration policy documents?
      There are a multitude of different policies required for a typical migration to run smoothly, it pays to agree these in advance of the migration so that the project initiation phase runs effortlessly.

    May 7, 2009

    Reducing IT Costs through MDM

    Filed under: Data Quality — Tags: , , — Olga Belokurskaya @ 5:08 am

    Today MDM becomes trendy, though not everyone fully understands what it really means for an organization. However, integrated MDM plays an important role in reducing compliance risk due to a lack of data transparency and trust, especially in the financial industries. In general, MDM aims to provide consistent, comprehensive core information across an enterprise, enables companies to realize internal efficiencies by reducing the cost and complexity of processes that use master data.

    I’ve recently found some ways, how MDM can reduce costs:

    • MDM centralizes common information, thus simplifies expensive business processes that rely on point-to-point integrations.
    • Eliminates duplicate, redundant data feeds provided by third-parties.
    • Allows cleanse all data across the enterprise thanks to integration of data from disparate applications into a single system.
    • Centralizes data across the enterprise which helps reduce the amount of redundant data storages and systems. Thus cuts the cost of needless licenses, support and hardware.
    • Configurable off-the-shelf MDM platform replaces antiquated custom applications saving significant development and maintenance costs associated with band-aiding custom solutions.
    • Eliminates reporting errors, expedite audits and improve compliance by
        a) managing a single version of the truth along with a history of all changes, and
        b) delivering this information to any reporting, business intelligence or data warehouse.

    However, one should not be overoptimistic about MDM and set his/her hopes on immediate results. There is a significant amount of work to be done to make MDM implementation successful. But when the work is done, and it is done right, MDM looks pretty beneficial. For apart from cost-cutting, it’s able to improve the ability to share, consolidate, and analyze business information quickly, both globally and regionally. And it makes it possible to rapidly assemble new, composite applications (software that combines the elements of a business activity in a coordinated application and user interface) out of accurate master information and reusable business processes.

    March 18, 2009

    Quick Tips on CRM Data Migration Planning

    Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Olga Belokurskaya @ 8:26 am

    In my previous post i reviewd some points that could help to increase the quality of your CRM database. Now I’d like to touch upon quick tips on what to do if you plan your CRM data migration.

    Data migration from one CRM system to another can be quite irksome. Although adding new records to the new configured and ready to use CMR is pretty plain, it’s quite troublesome to shift your previous data into new CRM. Often before the data in your existing format is ready to upload into the new system, it requires a big amount of formatting, enrichment and cleansing. It’s “an inevitable evil” that comes with the migration process.

    What you should keep in mind while migrating your CRM data:

    Make sure you have an exact back up of all your previous data and the new CRM so that you could roll back to where you were if anything goes wrong.
    Check which additional data fields are compulsory in the new CRM and identify them with the fields you have in your current CRM.

    Add any additional data items that are missing, remove those that are not required and make sure you have complete records which are ready to be migrated to the new CRM.

    As soon as the data moved to the new system, categorize and label it. Do it systematically to avoid the mess and have the retrieval easier.

    The whole process, of course, requires a lot of effort and quite dull work, but if it done well, it will be worth every spent minute.

    March 13, 2008

    Thou Shalt Migrate =)

    Filed under: Data Migration — Tags: — Alena Semeshko @ 5:02 am

    Back to data migration talk. Although DM might be associated with significant efforts and costs, it’s still better than leaving everything as it is (in a mess).

    Here are some reasonable reasons to migrate:

    • Increased compliance
    • Improved functionality
    • Future proofing
    • Reduced total cost of ownership (TCO)
    • Performance, reliability, scaleability
    • Reduced risk of scope cutting, cost overrun, or project delay
    • Improved data consistency across systems, processes, and organization
    • Increased responsiveness to the business

    March 12, 2008

    Data Migration Talk

    Filed under: Data Cleansing, Data Migration, Data Quality — Tags: , — Alena Semeshko @ 4:37 am

    CleanUp! You first hear these words as a kid from your parents. Clean Up! When you hear this you usually know you’ve made a mess. Clean Up! This is what you shouldn’t be hearing, or, for that matter, thinking, in regards to your company’s data. Or, at least, if the prospect ever crosses your mind, it shouldn’t look as nasty and unpleasant as it used to in your childhood. =)

    But nonetheless, clean up you should. If your source systems and initial data are a mess, of course. The obcession with clean data is only justified in this world of Business Intelligence, where looking at the picture as a whole and thinking big is not an encouraged, yet infrequent occurance anymore, but a requirement.

    One of the key elements to having your data clean and having a global view of your organization’s lifecycle is data migration. Wise data migration with an appropriate strategy and the right tools, not the sort where you splash money and remain in the same spot you started.

    Anyway, a whitepaper I came across got me thinking about this, so you can download it and check it out for yourself over here. It’s called The Hidden Costs of Data Migration and it touches upon the issue of data migration, whether to employ it or not, and the costs associated with it.

    Data migration has become a routine task in IT departments. However, with the need for critical systems to be available 24/7 this has become both increasingly important and difficult. This White Paper will outline the factors that are driving data migration and examine the hidden costs that may be encountered when data is moved.

    March 10, 2008

    Data Migration Pro

    Filed under: Data Migration — Tags: — Alena Semeshko @ 2:56 am

    While reading the news this morning, I stumbled upon a link to this new website called DataMigrationPro. Apparently, the new web site is devoted entirely to data migration!

    Thoughts upon checking it out: although it’s new and there isn’t too much user activity going on, I think it has enough potential to become a useful data migration database. Looks like a great opportunity for professionals to share knowledge, find relevant information and enrich their networks. There are interest groups and blogs focusing on the key areas of data migration, top data migration-related news and events, and much more.

    Data Migration Pro is a global community platform that enables members and organisations to deliver more successful data migration projects by enabling knowledge, opportunities and best-practices to be shared and developed.

    Membership is free and we welcome registration from all professionals and organisations connected with the data migration profession