CleanUp! You first hear these words as a kid from your parents. Clean Up! When you hear this you usually know you’ve made a mess. Clean Up! This is what you shouldn’t be hearing, or, for that matter, thinking, in regards to your company’s data. Or, at least, if the prospect ever crosses your mind, it shouldn’t look as nasty and unpleasant as it used to in your childhood. =)

But nonetheless, clean up you should. If your source systems and initial data are a mess, of course. The obcession with clean data is only justified in this world of Business Intelligence, where looking at the picture as a whole and thinking big is not an encouraged, yet infrequent occurance anymore, but a requirement.

One of the key elements to having your data clean and having a global view of your organization’s lifecycle is data migration. Wise data migration with an appropriate strategy and the right tools, not the sort where you splash money and remain in the same spot you started.

Anyway, a whitepaper I came across got me thinking about this, so you can download it and check it out for yourself over here. It’s called The Hidden Costs of Data Migration and it touches upon the issue of data migration, whether to employ it or not, and the costs associated with it.

Data migration has become a routine task in IT departments. However, with the need for critical systems to be available 24/7 this has become both increasingly important and difficult. This White Paper will outline the factors that are driving data migration and examine the hidden costs that may be encountered when data is moved.

In this world of changing business requirements, flexibility is the key to providing a positive customer experience and, in the long run, succeeding. Flexibility, however, is a complicated deal. Not all companies have the power, time and resources to constantly readapt to the new OC’s, new security measures, new software, new systems, updated and improved work strategies… if you spend all your time on these processes (and they do have the potential to take ALL of your time), who’s gonna do business for you?

This is precisely the reason why the SaaS market has exploded in the recent years and why companies increasingly put their trust along with their confidential data into the hands of outside service providers, rather than in those of their in-house staff.

Google’s recent survey on “message security and compliance”, involving 575 IT professionals, showed that the majority of respondents (53%) believe that “IT holds ultimate responsibility for their organization’s communications security and compliance. They also realize that organizations need to have policies and mechanisms in place to help achieve this.”

The survey confirmed the growing need for software-as-a-service (SaaS), “which is taking the place of in-house solutions that tend to require significant internal resources to maintain. The results suggest that one reason SaaS is gaining momentum is that the problems it solves are top-of-mind for IT departments, such as spending too much time or money on troubleshooting and maintenance.”

“By using SaaS service providers, organizations can offload capacity challenges, get real-time updates, and benefit from the economies of scale of a large network.”

While reading the news this morning, I stumbled upon a link to this new website called DataMigrationPro. Apparently, the new web site is devoted entirely to data migration!

Thoughts upon checking it out: although it’s new and there isn’t too much user activity going on, I think it has enough potential to become a useful data migration database. Looks like a great opportunity for professionals to share knowledge, find relevant information and enrich their networks. There are interest groups and blogs focusing on the key areas of data migration, top data migration-related news and events, and much more.

Data Migration Pro is a global community platform that enables members and organisations to deliver more successful data migration projects by enabling knowledge, opportunities and best-practices to be shared and developed.

Membership is free and we welcome registration from all professionals and organisations connected with the data migration profession

As I mentioned in the previous post, data cleansing deserves a post of its own. Even more than just one post actually.

Well, it’s obvious data is the key player in business decision-making. Good clean data provides the platform for wise decisions that put the company’s profits onto an upward curve.

Acquiring the right data, however, is not always as simple as it seems. The techniques are many, but the effect from them doesn’t always meet the expectations. That’s where data cleaning technologies come in place. Data cleaning software cleanses the initial data, making it more precise, useable and up-to-date. Techniques used in data cleaning, among others, include:
• Data merge from data sources
• Record matching and synchronization
• Data type and format conversion
• Data segmentation

In this post I want to focus more on record matching and data synchronization.

An example that is often used in this regard is name and address data. Name, address and phone information is the quickest to get outdated and easiest to get wrong. Of course, there are directories and yellow pages that you can always check…but if you do it by hand each time you encounter a mistake, that’s an impermissible luxury in that it takes way (I mean waaaaaaay) too much time.

That’s pretty much the reason and the root of data synchronization technologies. They process the data, compare it to the standard and return a valid quality dataset with all possible mistakes (misspellings, wrong street type extensions, city and state names) eliminated. Apatar’s StrikeIron US Verification data quality service, for instance is one of such tools.

Employing sophisticated matching and data synchronization technology, it first closely inspects each address to ensure its validity and then updates incorrect addresses according to postal standards and cleans customer data before it gets into CSM/ERP systems, databases, flat files, and RSS feeds. It also adds ZIP+4 data, specifying congressional districts, carrier routes, etc. Data cleansing tools of this sort are indispensible in business today. They allow companies to increase productivity, improve sales strategies, and deliver a better and more accurate customer service.

ETL (data Extraction, Transformation and Loading) is an integral part of data warehousing.

In brief, ETL processes are used to extract data from various sources, transform it by cleaning and integrating, and finally load it into data warehouses. The data you get in the end is clean, well-structured, systematic, and ready to use. It sounds quite easy, but in reality there’s much more to it. There’s a catch to each step.

During the first step, data extraction, the challenge you are dealing with is the bulk of data from different sources. It might be different departments, databases, formats, reporting systems, etc. This scattered data needs to somehow be captured and moved into the staging database.

Next — data transformation. This step is the most complicated. It can be broken into four separate steps under the umbrella of transformation:
- data verification – comparing the extracted data with the DW quality standards. In case data doesn’t meet the outlined standards, it either gets rejected, or held to be reviewed by the administrator.
- data cleaning – the data left from the previous step is made more precise. (The techniques this stage includes are so many that they deserve a different post.)
- data mapping – merging data from different sources into a single interface, structuring it into columns and tying it together logically.
- data consolidation (or aggregation) – summarizing data from the previous step and performing overall calculations to provide the user with a more complete picture.

Finally, loading. This step simply uploads the data organized during transformation into a warehouse.

During this whole process, the one thing to be careful of is losing data. ETL process does not presuppose changing the initial data, it should only make it better, cleaner, more correct and organized.

Back to yesterday’s post on BI. With Data Warehousing being an indispensible attribute of BI, I’d say it’s also one of the key components in making the company’s decision-making lifecycle more eficient and productive.

In the world of Business Intelligence there’s no place for people who manage by gut. Auch that hurts, huh? But that’s true. People who use their intuition or gut feeling to make major decisions in business usually lose to those using BI in support of management decisions. It’s like with cars: your serviceman knows exactly what that noise under your hood means and what has to be repaired or replaced in your car, while you might only suspect that something’s wrong with the engine or breaks or gearbox and if you were to repair your car, you’d be more likely to break something else than repair what’s broken. Employing BI strategies and techniques, like, for instance, data warehousing, provides the security and assurance you need to keep your business up and be sure of your decisions.

When success depends on how quickly a company responds to rapidly changing market conditions, BI is where you turn for help. It fast-forwards the decision-making processes and provides you with the insight necessary to make the right decisions faster.

With the modern technologies of data integration, warehousing and analysis, you get a single complete view of your organization’s past, present and potential future with the major problematic areas already figured out for you. All that is left is for this perspective to be put into action.

*get going*

Scott Davis has recently blogged on the idea that reduction does not necessarily mean simplification. As an example, her used his son’s physics class, where kids were more comfi with a loger formula than with its shorter equivalent. It was easier for kids to comprehend and intuitively solve the longer formula as “each equation component was visible and was easy to relate to each of the respective forces at play”. Scott goes on discussing this issue by taking it into today’s business world.

“We try to boil down lots of unique moving parts of the Strategy Gameboard into a single pithy Vision Statement — and then, we’re surprised when employees feel no connection with it. Or, we try to homogenize data and reporting from the variety of functional areas around the business into an non-descript blob of a common-denominator centralized Business Intelligence program — and then, we’re surprised when departmental analysts keep using their own stuff.”

You know…when I was in high school I always prefered shorter and simpler equations. It was much easier for me to just memorize a few things and forget about the lengthy process of figuring what goes where each time you have to solve something in Physics or Chemistry class. To me it was way easier to make a mistake when solving extended equations.

Now, I understand, not everyone’s like me and some people are more comfi with taking their time, getting to the root of everything. Maybe. Although in the words of business, particularly the constantly on-the-move business today, that’s not at all the best approach. What use are employees that take twice longer than others to complete a task? They might be as professional and skilled as others, but… let’s face it, we live in a rapidly progressing competitive world. To succeed in this world, or at least to secure your place, you’ve got to work and think fast.

Same goes for software. If earlier software programs were designed to address specific needs and run independently, today the market for packaged enterprise applications is exploding, creating an emmediate need for integrating multiple independently developed and heterogeneous applications. Simplification, integration, reduction of aplications running.

Same goes for data. Enterprises today tend to replace their passive reports and charts with active business platform based on adaptive IT architectures. For better or worse, business intelligence tools enhance business communication across enterprises, coordinate resources, and enable companies to interact more quickly in our ever-changing world.

This “reduction”, which is more of progression and interconnection actually, is not seen as negative. Quite the contrary - it’s critical to today’s businesses

The use of business intelligence software has now assumed a high profile in many organisations, says the research, with 66% of senior managers seeing it as “very critical” or “critical” to their decision making.

This positive approach is reflected in the fact that 67% of organisations plan to extend the scope of their data warehousing activities to new functional areas, while 64% will be further exploiting their existing information.

Well, the numbers speak for themselves =).