Project X Discussions

Rushing to Solution

One of the things I noticed in myself and others is how quickly we rush to solutions.  In a business intelligence project, rather than finding out the business questions the client has, we rush to designing reports.  In the process we narrow our solution much to quickly.  If we explored more fully the questions the client might have the solution might be able to answer many more questions than the initial report we design which will then reflect how we structure the underlying data. 

I think one of the difficulties is thinking in a multi-dimensional world.  The exercise of thinking of information organized by multi-dimensions is difficult for people to visualize.  However a report format is relatively easy to visualize.   

However the report is not the only problem.  Most of us in the IT business are great problem solvers and love to solve problems.  Unfortunately we sometimes are already designing a solution before we really understand the business requirement.  In addition the client thinks of his issue in terms of what kind of report or spreadsheet I need.  Unfortunately a report or spread sheet in two dimensional. 

I think before we rush to designing reports, we should work on a logical data model which deals with the multidimensional nature of the business problem and questions.   We can consider how these things fit together.  The next step would then would be to think about how to answer the business questions. 

How many of us have rush to report layouts or demo reports before we understand the business problem?  I think we are so attracted by the lure of the technology we sometimes end up in the wrong place.  However using agile methods can really help us avoid getting stuck on the initial solution.

I do not think there is any simple solution to the lure of rushing to solution.  The main idea of this blog is to make us all more aware of the temptation to rush to solution.  Let us all resolve to understand the business questions more fully and help our clients think about questions and not reports.

Database Update Delay Cost a Life

An article in the St. Petersburg Times on March 1, 2008, caught my eye entitled "Database Delay may have hindered search"  The article says the updating of a state database may have kept police from finding a mother of two alive.  The policed focused on a green Camaro and the police checked three nearby Camaro owners.  But King, who registered his car a month earlier, was not in the database, which is updated every 90 days.  The state wants updates to be weekly. 

Clearly the Extract Transform and Load (ETL) process needs to to much more efficient or the system needs to be updated to a faster process.  Some of the ETL processes can be very cumbersome.

I think of Graham's discussion about getting the transaction and inquiry database integrated.  Could save lives and prevent lots of fast moving criminals.  If the process was efficient, why not have the update close to online? 

CoffeeCAST #52 - Consultant in the Middle

Welcome to the 52nd CoffeeCAST with Stephen Hayward of Project X Ltd.

Happy New Year and we are back on the air as it were.  I have asked Graham to join me while we talk a little about:

1. Consultant in the Middle / Readiness

1. Master Data Management (MDM - IBM, Oracle and Teradata)
2. Moving data around costs money
3. Business Intelligence Trends in 2008

Download 52_coffeecast_52_consultant_in_the_middle.mp3

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.

Data Definitions - Wait Times

If one is going to use data to make decisions or measure performance, definition of data is very important.  I had a discussion recently with a friend about "Waiting Times" in the delivery of health services in Canada.   The reason for this discussion is I get many questions from my American friends about whether the health care system in Canada works.   I pointed out to the dilemma we have with the way our system works.  The Federal Government provides a lot of money to the provinces to deliver health care, which is a provincial responsibility.  However the Feds want to influence the delivery and made a condition of some of the money, reduction in "Wait Times". 

Now the fun begins because now we must define "Wait Time".  Lets us take a simple example.   I go to an emergency department with a broken leg and in pain.  I get service in thirty minutes.  Then another time I go to emergency because I have a bad cold and it is on the weekend.  I wait for eight hours before I am seen by somebody who can give me an examination.  I think it is obvious we cannot consider these wait times in the same category. Similarly elective and emergency surgery and not the same.  So we now need data definitions that all provinces and the federal government agree on.

Then we can start collecting meaningful data and evaluating the performance of our health care system in some meaningful ways.   Unfortunately the issues are complex and people want simple models. 

I intend over the next few months to do some research on the issue of data definitions with "Wait Times" because I think it demonstrates the importance of data and data definitions for business intelligence.  I should I say for businesses to act intelligently.   

If we have bad data because of poor data definitions, then we will likely make bad decisions, based on bad business intelligence.

Innovation - Time for Some Recognition

Well, it is not often you get an award from a client.  Well thanks to some hard work by Doug (one of our Senior Consultants) we/Doug received recognition on some work that was done.

The Recognition - "Exemplifying Breakthrough Innovation"

Well aint that a peach.  We are not allowed to talk about what Doug and the team he worked with did, but suffice it to say they turned a competitive situation for a client on its head to meet a business threat.

By focusing on the objective he worked within the system to do what normally would take months in less than two weeks with only one defect so far.

Great work Doug and the team.

Extending the Enterprise Data Warehouse (EDW)

One of the challenges in the EDW world is extending the data warehouse.  If you are adding new data to the warehouse, you will need likely to change the model or at least update the schema and some other metadata.   So what is the advantage for the sponsor of the project to add it to the EDW?  The temptation is to add a working data store which is related to the EDW but not totally integrated.  Certainly not as elegant but can be implemented more quickly.

I expect as a user of the EDW and benefiting from what it offers, the sponsor likely has some obligation to make the new information available in the EDW.  In addition the response time with the integrated data is likely much faster and in the overall easier to maintain.  In the longer term, integration allows you solution to evolve with the EDW. 

My suggestion is to first implement a working data store solution and get payback.  If there are advantages in integrating the data into the EDW, then a separate case can be made.  The difficulty of the case is that maintenance is one of the hidden cost of a working data store.  However I think those trade-offs need to be presented upfront and then later revisited as we proceed. 

Another possibility would be develop a phased approach to the project that include the both a working data store approach, a refinement of the design to better meet the broader EDW requirements and integration.  All these steps could be outlined up front and decisions made about the correct approach. 

The ideal EDW is one with a completely integrated system that respects all the fundamentals of EDW design.  We need to aware of the ideal but recognize the realities of the business. 

 

Updating the Enterprize Data Warehouse (EDW)

An interesting challenge arises when changes are being made to the EDW.  If the current requirements are to remove some data that is no longer valid.  For example a product or service is being removed and so data which was being extracted is no longer available to be extracted.   The question is do we remove the historic data from EDW or simply remove the data structure from the updating process and keep the historic data.   The current structure has now different form the old data structure. 

I think to maintain the integrity of the EDW, the old data should be preserved.  However if maintaining the out of date data cost more, who should pay.  An interesting architectural and philosophical issue. 

 

A Burning Issue for Data Warehousing

I cannot resist quoting Roman on this issue:

"Question: Data warehouse as a set of consolidated data marts or the “real deal” unified enterprise data model? 

Answer: A person capable of devising a unified enterprise data model, which will explain all in the enterprise and be accepted by all in the enterprise, will also be capable of devising the “Grand Unified Theory”.  Will that be God?  Not sure.  Will the person be treated as God?  Yes".

Roman Turezki,

Jim's response:

How about calling it the Grand Overall Data (GOD) or Grand Overall Design?

Challenges of the Business Analyst

The work of a business analyst is to develop an understanding of business process and model them.  Usually the work is associated with a project whose objectives are to change or improve a process.  Often these processes are quite complex and the analyst must get the information from many sources.  Usually much of the information and ideas for improvement are in the heads of key users of the processes being studied.  The challenge of the analyst is to get a good understanding of the process from these people. 

This task presents the analyst with many difficulties.  A business analyst often encounters people who are not ready to cooperate for many reason. 

• Lack of understanding of the project
• Impatience, "I've told four other people the same information."
• Fear of having their job automated
• Fear that their lack of expertise will be exposed
• Opposition to the project
• Previous bad experience with analysts
• Person is already overwhelmed by their job, no time.

These can be summarized by saying the person is not ready to cooperate.  So how should the analyst proceed. 

I have found that the first step in any meeting between people is the development of rapport.  The people need to feel comfortable with each other as people.   How many times have you been meeting with somebody and things go very poorly and you had the premonition in the first twenty seconds.  Likely no rapport has been established between the people. 

Once rapport has been established it will then be possible to address the concerns of the person.  If you try to address the concerns without rapport you will be like ships passing in the night.

Establishing rapport can be thought of two people getting on the same wavelength or feeling comfortable with each other.  Building rapport is a skill that can be developed and requires some work. 

You need to have some information about the person you a going to meet with.  If you have mutual acquaintances, you might get some information about the person's likes and dislikes, job history, personal situation, interests, etc.  The idea is to find something that you share in common that is not controversial that you can talk about.  Often people will talk about the weather or share a cup of coffee which is a start but often not enough.  Often if you are in the person's office, you will see pictures or other decoration which reveals their interest.   Often something related to the business or people you know in common is helpful but remember the subject is something you will agree on.  Politics and religion are not safe ground.

Having established rapport, the next step is to be clear bout the purpose of the meeting and get an agreement on the agenda, including the time frame.  Here is where you address any concerns that you detect. 

Although you will be anxious to get down specific details, I strongly recommend to stay at a higher level for a while.  Ask open questions that find out the person overall view of the project and how it fits in with their interest.  An open question like "Tell me what you think about ..........?" or "Could you give me a little of your background?"  Once you have some context for the information, you will be much better evaluate the answers knowing some of the biases of the person.

Often the person's reluctance will stem from little or no understanding of the project.  Without a context the information you glean from the person may be totally off the mark and misleading. 

If you have started the conversation and you feel that you have lost rapport for some reason.  Gradually re-establish rapport.  One technique I use is to summarize where we are so far and get agreement on that.  If you detect, reluctance, ask "Do you have some concerns?"  Keep the question very open and listen carefully what the person says.

Another problem that arises between systems people and the rest of the world is that we use a lot of jargon and short forms that are foreign to others.  Using those words, reminds the person we are in different world or world apart.  NO RAPPORT.   Listen to the words the person uses to describe something and use the same words.  If they have a name for a function, use that name.  If they have a name for a process, use that name.  Stay away from the technical jargon we all slip into in the IT world.  "The ETL process of EDW is critical to Micro-strategy software.  We need to understand how to normalize your data so we can drill down and across."   If you thought I was from the moon before, I just moved to Mars.

I will write some more about business analysis later but remember stay in rapport.  You will be amazed how quickly you can get the information you need when you are in rapport. 

 

Data Warehousing Fundamentals # 3

I think a little history of data warehousing (DW) would be helpful to understand how it has evolved.  In the late 80's and early 90's the idea of an executive information system (EIS) started to be the fashion.  Programs were developed that would present data in a very user friendly way on executive's desk.  This data might be key performance indicators and lots of other external and internal information.  Tools were required to extract data from all kinds of different sources and put them in a form that executive could use to explore the data.  I remember one group develop a tool called the Executive Dashboard.  That was the first time I heard about drilling down into the data. 

People discovered many challenges around the extraction of data and how to put it into a form that made exploration quick and easy.   Ideally the data would be from live data so that information was not replicated.  However this idea proved difficult because performance of operational data bases was a big enough problem without adding a whole new set of complexity.  Thus the idea was to extract data from the operation data bases and transform them in data suitable for the EIS tools.   

Several data base organizations came out with tools to meet these needs.  These new executive data bases became repositories of all kinds of really important data about what was happening in the business and in the business environment.  For example a mining company executive could track the price of a metal on the open market with the price they were getting for the same product and compare both of these to their production costs.  They could do this monthly and watch trends.  Some of these executives became very sophisticated in using this data to make important decision.  One organization developed a very sophisticated tool to measure the profitability of every sale versus the price they could get on the London Metal Exchange.  This data was very close to real time and I recall was tracked daily.

Eventually people realized this data was a important asset and the data warehouse concept was born.  Some companies realized huge competitive advantage by using the internal and external data.  The accent is legendary in the data warehousing lore.  The problem developed that the size and complexity of the DW grew exponentially but processing capability was lagging.  Some companies developed specialized machines that could handle these demands.   Thus response issues were address.  However these machines and the data were still not being used by the people who needed the data for running the operations.  The data was still extracted and transformed into a format for the DW.

We are now getting closer to being able to use the same data for operational and reporting requirements.  Clearly this would be better because every time data is duplicated the cost rises and the chances of error increase.  This design goal presents many challenges for the system architects and designers.  However the dream of data base design is to have a data entity only exist in one place and be related to other data by pointers.  These type of data bases are called relational data bases.  Often this ideal is sacrificed for performance.   More on this subject later.