Project X Discussions

Data Warehousing Fundamentals # 2

One of the design fundamentals in data warehousing (DW) is ability to drill down.  The term is really a part of DW jargon so I would like to explain it more fully.  I would like to use an example of a retail store which keeps data on products sold, customer, purchase price, date, quantity purchased, total purchase price.  Each one of these categories of data is called a dimension.  If you would like to know how many customers bought this product on a particular day, you need to drill down into the details in that dimension.  Then total up the quantity sold.  That is a simple case of drilling down.

Now drilling across is different and the idea is that you move across dimensions.  If you would like to know total sales on a date for all products, you would add sales on that date.   That is simply drilling across.

Another way of thinking about the data being stored in matrix with many dimensions and data being stored in each dimension.  A simple two dimensional matrix of customer and product is simple to visualize.  However visualizing the huge numbers of dimensions common in a business is very difficult.  I think this complexity makes it hard for businesses to keep all the complexity of their business their mind.  They then create a simpler model of key success indicators.   The DW could then report on these key indicators.  However when one of these indicators is awry, we then need to drill across divisions or departments to find out which part of the business is causing the problem.   When we find the division that has a problem we need to drill down into the division to find the source of the problem.

I suggest you try to develop your own mental model of what all this means for you in your business and your role in the business.  I think developing a good mental model of these complex processes is very important and reviewing the fundamentals will help develop these models.  The big difficulty is not resorting to jargon as we try to communicate these ideas.

 

Requisite Organization

I am reading a book now called the Strategy Paradox by Raynor, which has some fascinating summary of other people's research, as well as his own ideas.  One person he refers to is Elliot Jaques (pronounced Jacks), who did lots of good research into management, and really is not recognized for his contribution.   I remember trying to read one of his books some time ago and becoming quite confused.  I do not think he expressed his ideas well.  Jaques did a lot of research on organization structures.

One of the interesting bit of research he did on organizational hierarchies revealed that people at every level have two bosses, one their boss and the other their "real boss".  The first is the person they were nominally responsible for their actions, the latter was the person they could get a decision that matter in their work.  He found this level skipping went both ways.  The people higher up found themselves going around their immediate reports to get relevant and timely information.  This activity makes the organization dysfunctional but he found that this activity was very common.

During his study, Jaques worked on how to distinguish between different jobs in the hierarchy.  He discovered that jobs could be defined by their time horizon.  One person could be concerned about this months results, another the quarter, another the year and others multi-years. 

He found that people who had a time horizon of less than three months, considered the person with a time horizon of three months, their "real boss".  In his work he identified seven levels of hierarchy based on time horizons no matter the complexity of the organization.  He called the seven levels, Requisite Organization.  Often companies have extra layers of organization.  Jaques calls them integrating layers that have the same time horizon as the the layer that reports to them. 

In order to verify his theory Jaques took information for studies done "felt-fair" pay studies.  He found that the criterion of time horizon for decision making was the criterion that had the highest correlation with feeling fairly paid.  Other criteria for feeling fairly paid, like know-how, job impact and seniority, were dramatically less correlated.

This all very interesting and we can all recognize weakness in organizations where we have work.  However all it does is describe the organization and predicts difficulties.  What the theory does not do is tell us what to do about the problem.  I think this fact and the way Jaques expresses his ideas kept his idea becoming mainstream.  Even the name is a bit pompous. 

However the idea of using time horizons to analyze organization decision making and the setting of strategy can be very helpful.  The pace of change in technology is changing the time horizons of all levels of management. 

Multi-year projects to introduce new processes and technology have huge built-in uncertainty.  The people who have to make these decision live in a world of uncertainty.  The author of the Strategy Paradox coins a term "Requisite Uncertainty" to adapt Jaques idea to peoples planning horizons.  The longer the time horizon, the greater your range of choices, but less certainty of what precisely to do. 

The shorter your time horizon is the more important it is to have a strong commitment to your course of action because your options are considerably less.

Thus Jaques theory help us think about the roles of various levels in the hierarchy in the strategy of an organization.   I also find it helps me model organization structures that might be a bit confusing.

I expect the tension between an IT organization and their users can be traced back to time horizons.  That is why we feel that a focus on Rapid Results in the context of a broader strategy and this focus helps bridge the gap of time horizons. 

Complex Project Sponsorship

Often in IT Projects, the sponsorship of a project is not simple.  Often there a many stakeholders who have a significant interest in the project.  This is common when we are working on a data warehousing project.  Many different groups have vested interest in the project.  These interest might also come in conflict.  In particular problems will arise if one group is responsible for the requirements and another for the budget.   What are some actions you can take to mitigate the problems?

My first advice is to try to simplify things.  Harvey used to say "I am a simple man so you must make it simple for me."  That means in your mind deciding who really cares and make sure they are supporting your activity.  They will go to bat for you and clear away obstacles.  In addition some of the people have more influence and power in an organization.  I have found it does not necessarily follow from the organization chart.  Some people just know how to get things done.

I have also taken on a role of communication medium between all the various interest especially if the group do not meet with each other for various reasons.  Often groups have history which inhibits good communications and trust.  By focusing on the objective of the project and repeating it frequently sometimes you can get people all looking in the same direction.  I often visualize the objective as something which you can point to on a wall or in the centre of the table and metaphorically getting them all to look the objective as oppose to each other.  Shuttle diplomacy really works. 

Another thing to remember is that nobody means to do the wrong thing.  It may appear wrong to you, but from their perspective it is right.  The problem is to see it from their perspective.  I often become very confused and admit it.  I then asked for help in joining the points along a logical path.  Often the explanation will clear up the confusion and you have better idea why the apparent conflict exist.  Then you can set about getting the players to focus on the goal.

Usually complex projects means that many people in an organization have to change the way they have always done things.  Some people may not even understand why they do things they do but just know it works.  For them to step into the unknown is scary.  These people need support and help from the people they trust who can help them make the change.  Often simplifying things will help because many people do not need to understand the complete system just their part.   Make it simple and low risk.

We all know on complex projects not everything will exactly as we expect even with extensive testing.   By letting people know that things may not go exactly as they hoped and giving them ways of getting help is important.  Eventually as they gain experience with the system they will learn the things that do not work well and avoid them. 

A real key to success of a project is for people to know that the project is important to their "real boss".  A note or email from the project sponsor or sponsors telling people how important the project to the sponsor and thanking them in advance for their support will help.

I am sure there are lots of ideas and experiences people have.  Like to hear some ideas.

Definiteness of Purpose

Many years ago I read a book or listened to the tape of a book called "Think and Grow Rich" by Napoleon Hill.  It is a book written in the 1930's.   I discovered the book and Hill listening to tapes by Anthony Robbins where he interviewed authors of his favourite books.  So, although Napoleon Hill was dead, he had recorded many interview so Robbins took clips from these interviews and fabricated an interview.  It was fabulous and so I got the book tape and listen to it in my car as during my commute.  Hill was talking about the Law of Attraction in his own way.  One of the things he emphasized in his own quaint accent was "Definiteness of Purpose". 

The message was the things we give our attention, energy and focus are the things we attract to our life.  I recall at the time there was a lot of upheaval in my life and I have some concerns about my long term prospects.  I realized that my income was completely determined by my time and I needed to change that.  I therefore created a definite purpose without really knowing how I would get from there from where I was currently. 

Well, circumstances changed dramatically for me and I was able to accomplish my goal over the next fifteen years.  Looking back I realize that was a real turning point.  As time went on I was able to attract more opportunity to accomplish my purpose.

So recommend that we all have a definite purpose in our life, write it down and allow it to happen.  As things happen to align with your purpose, celebrate them and continue on the journey.  If doubts creep in, remember "lots can happen" and celebrate how the journey has gone so far.

The book or tape of "Think and Grow Rich" is really unusual but contains some good lessons for life.  I was certainly ready at the time and the teacher came.

La querelle du regionisme au Quebec (1904-1931) Part II

The Canada Council for the Arts announces finalists for the 2007 Governor General’s Literary Awards

Non-Fiction French

Annette Hayward, Kingston, for La querelle du régionalisme au Québec (1904-1931): Vers l’autonomisation de la littérature québécoise
(Éditions du Nordir; distributed by Prologue) (ISBN 978-2-89531-049-5)
This work marks a significant step in studies on the literature of Quebec in the first half of the twentieth century. With a rigorous approach and a precise, elegant style, the author allows us to relive a major episode in the literary life of Quebec.

See La querelle du regionisme au Quebec (1904-1931) blog

http://pxltd.typepad.com/project_x_discussions/2007/05/la_querelle_du_.html

See also book web site

http://querelle.didascom.com/

CoffeeCAST #49 - Teradata Partners 2007 - Wednesday Closing General Session

Welcome to the 49th CoffeeCAST with Stephen Hayward of Project X Ltd. Live from Las Vegas.

Hello again from Vegas - baby.  We are coming close to covering the end of a great week at the Teradata 2007 Partners event.  The closing session was great, Graham and I continue with some discussion on the closing session.

This was an inspiring session which had the theme of innovation.  So before we go on to our last CoffeeCAST in regards to this event, here is the discussion that Graham and I had after the session by Dean Kamen (most notably known for the invention of the Segway, but the least of his advances)

1. Innovation versus Invention
2. Innovation is more about surprises (negative) what he refers to as rude realities of issues which are unforeseen
3. Seven Rude Realities
4. Three Suggestions
5. Introduction to Deka Research and Development Corporation
6. A very fast overview of FIRST - For Inspiration and Recognition of Science and Technology a great and fast growing movement to bring the 'bling' into Science and Technology

Download 49_coffeecast_49_teradata_partners_2007_wednesday_part_1.mp3

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.

CoffeeCAST #48 - Teradata Partners 2007 - Tuesday Sessions

Welcome to the 48th CoffeeCAST with Stephen Hayward of Project X Ltd. Live from Las Vegas.

Hello again from Vegas - baby.  It was the end of a great week at the Teradata 2007 Partners event and Graham and I continue on to discuss more of our thoughts on the conference.

Tuesday was a jam packed with some great stuff.  So we summarize at a high level by talking about today's sessions that I attended:

1. Future Trends in Data Warehousing and Business Intelligence - Gartner Presentation by Donald Feinberg
2. What's Your Strategy for Master Data Management - A discussion on systems convergence
3. Gaining CxO's Support for your EDW and Business Intelligence - Ron Swift - Teradata
4. Teradata Active System Management (TASM)
5. A New Era of Data Integration with ELT approach - CTO one-on-one with Stephen Brobst of Teradata
6. Active Ingest to Support Active Enterprise Intelligence
7. Back Up And Recovery - early morning session
8. What's New with Stored Procedures
9. The New Query Rewrite Subsystem in V2R12.0
10. Table Functions: A Tutorial

So as you can see a lot of covered ground, so we only go into it in a couple of areas.

Download 48_coffeecast_48_teradata_partners_2007_tuesday.mp3

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.

CoffeeCAST #47 - Teradata Partners 2007 - Monday Sessions Part 2

Welcome to the 47th CoffeeCAST with Stephen Hayward of Project X Ltd. Live from Las Vegas.

Hello from Vegas - baby.  It was the end of a great week at the Teradata 2007 Partners event and Graham and I decided to take some time away from the wine and tables to discuss our thoughts on the conference.

Monday was a busy day with some great stuff.  So we continue by talking about today's sessions that I attended:

1. Performance Management
   • Top 7 Best Practices
   • The Cause of Poor Performance
   • Strategies and Approaches
     • Data Collection
     • Performance Management
     • Planning
     • Workload Management
   • Tools

Download 47_coffeecast_47_teradata_partners_2007_monday_part_2.mp3

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.

CoffeeCAST #46 - Teradata Partners 2007 - Monday Sessions Part 1

Welcome to the 46th CoffeeCAST with Stephen Hayward of Project X Ltd. Live from Las Vegas.

Hello from Vegas - baby.  It is the end of a great week at the Teradata 2007 Partners event and Graham and I have decided to take some time away from the wine and tables to discuss our thoughts on the conference.

Monday was a busy day with some great stuff.  So we start by talking about today's sessions that Graham attended:

1. Teradata Query Optimization
   
   • Statistics
   • Evenly Distribution across nodes
   • Indexes
   • A Little discussion on Nodes and Amps
2. 20 Tb/hour Backup
   • Backup and Recovery
   • Netvault from Bakbone Software
   • The story from eBay
     • Custom built
     • Sun hardware
   • Leveraging Archmain
   • After Image Journaling

Download 46_coffeecast_46_teradata_partners_2007_monday_part_1.mp3

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.

CoffeeCAST #45 - Teradata Partners 2007 - Keynote Address

Download 45_coffeecast_45_teradata_partners_2007_keynote_address.mp3 Welcome to the 45th CoffeeCAST with Stephen Hayward of Project X Ltd. 

Hello from Vegas - baby.  It is the end of day one of the Teradata 2007 Partners event and we have decided to break up the week as individual CoffeeCAST (or as Graham would like to call this one WineCAST as we are at THEhotel at Mandalay Bay bar and not a usual coffee shop having a glass of wine.

Today we had a chance to hear some of the first messaging from Teradata since they went out on their own from NCR on Oct.1st.  So we start by talking about today's keynote address where we heard the following:

1. Mike Koehler - President and CEO Teradata - Raising Intelligence
   • The new Teradata allows them to Integrate, Manage and Use - some great strengths
   • Teradata 12
   • Dual Active
   • Being Green - 5500 server - 2.6 x increase in performance with 75% less energy consumption
   • Partnerships - Microsoft Analytics, Agilent and SAS (see below)
2. Announcement of SAS Partnership and Dr. Goodnight who got up on stage with Mike
   • Joint Engineering
   • Joint solutions
   • Joint Roadmap and Selling
   • Integration of the two products
   • Centre of Excellence for Architects and Consultants
   • with the Goal to cut the cost of data analytics
3. Geoffrey Moore - Analytics and Innovation
   • Why aren't analytics everyhwere - a brief discussion and point of view
   • Business Architecture Model
     • Complex systems - smaller volume hogher cost environment
     • Volume operations
   • Catagory Maturity Lifecycle
     • Tech adoption
     • growth market
     • mature market
     • declining market
   • Change the vector of innovation

Listen in and we hope you enjoy and more importantly, hope you join in the conversation.  So drop a comment in the blog or send me an email and share your thoughts on the topic.

The audiocast is available on iTunes (as a Podcast) or here for download.  Have a great day and join the conversation.