Project X Discussions

Big Data: The Next Big Thing

Big Data: The Next Big Thing

First a brief personal history of data warehousing and BI.

In 1987, I started working with relational database systems when they were in their infancy. We tried to build OLTP and OLAP systems on single relational databases, client server platforms. It didn't go so well. By the early 90s we'd figured out that the reporting environment had to be separated from the transactional environment if we were going to meet performance expectations. Inmon and Kimbal came along with their two views of the data warehousing world and datawarehouses for OLAP were replicated from the OLTP and legacy systems to do executive reporting. For about 20 years we've been writing ETL to populate the EDW, then pre-aggregating tables, and refining BI tools to build sexy reports and dashboard for people who don't know how to or can't do the work but need the numbers to make better business decisions. But that process still takes too long and won't handle all the data that is relevant to the business requirement to know and understanding our customers likes and dislikes so we can provide better service and ultimately entice them to spend more money with us buying our products and services.

Enter the latest hardware , super cheap servers with super cheap disk that can be clustered together into massive arrays of storage. Moore's law seems to be holding as all this technology just keeps getting faster and better and cheaper .

Mean while everybody who matters to marketers is connected in some way to the network that has evolved over these years. The Internet that no one knew what to do with 20 years ago has become a ubiquitous essential tool for global communications, enabling McLuhan's Global Village. We can text and tweet and Twitter to anyone on the planet in seconds. An impossible scenario even one human generation ago.

Now we can collect transmissions and transactions off of every chip, card and device that is connected to this ubiquitous network. All of a sudden we have the Internet of things. Machines that talk to each other without our aid. Collecting the data we generate. Some of us have a vague idea, but most of us can't even fathom the breadth of this data content and volume. And all this data to what end? Data that will allow us to monitor our homes, drive our cars, travel the world, design and build our better society. Enough data so that the marketers can predict what we are going to buy and when we are going to buy it before we even know ourselves. Brave New World meets Minority Report.

But there is so much data we need ways to parse it. Mine it. Searching, sorting, and tweezing out the nuggets that matter from the masses of html and snippets of text and escape sequences and smiley faced modicons. The separation of the valuable information from the ubiquitous noise. The wheat from the chafe.

I was recently in the Musee d'Orsay in Paris. Our guide took us to this painting :

http://en.m.wikipedia.org/wiki/The_Gleaners

After the main harvest is over, the peasants come in to the field to pick up the few grains of wheat that are left behind.

And that's what the Big Data world is all about. It's The Next Big Thing in the world of ETL and BI tools that will allow us to manage all the data that is coming at us from all the devices we have accumulated, and have become so dependent upon, in our Jetson like world.

Not much has changed in 150 years, we're still gleaning, just data instead of wheat.

data scientist 2

this is a second note on the role of data scientist... some links to

Harvard Business Review

http://blogs.hbr.org/cs/2013/01/the_great_data_scientist_in_fo.html

http://blogs.hbr.org/cs/2013/04/two_departments_for_data_succe.html

What's a Data Scientist ?

Twice this week I've encountered the newish title "Data Scientist" and have been struggling to understand what that role might entail.  The first place I came across the title was in an article in the Teradata online magazine:  

http://www.teradatamagazine.com/v13n01/Features/Data-Science--Future-or-Fiction-/ 

The second occurance was in an client's office where the subject came up and we speculated on the nature of the role.

If "Data Scientist" is the new, in demand, thing then as a provided of IT services I should understand what this talent actually is.   But, after reading the above article I was still at a loss to come up with a job description, skills inventory, or experience level that would qualify someone (like me or someone I know) as a "Data Scientist".   After a day or two I was still struggling for a definition, so I made one up.

Intuitively, we can split the title into Data and Scientist and derive a literal meaning.  Data: has extensive experience with the research, study and analysis of data and information in its many forms (structured, unstructured, human or machine readable, cleansed or not, fully qualified through metadata or not, modeled or not, etc.).  Scientist (picture lab coats and clipboards): someone who has an understanding of and expertise with the Scientific Method:

http://en.wikipedia.org/wiki/Baconian_method

This leads me to believe there would be basic research* and uncertainty associated with the role and its deliverables. The outcomes may not be known at the outset and there would be some risk that no value is realized from the process.  These Data Scientists would work in environments where solutions would evolve over time and change would have to be anticipated.  It also implies we don't know what we don't know at the beginning and therefore we'd be advised to proceed with caution. 

Alternatively, Data Scientist is a new definition for a Senior Business Consultant and Technical Data Analyst and Data Subject Matter expert.

My contrarian side fears it may actually be the role of a "Data Pseudo-Scientist" should Baconian methods not be implemented as rigourously as required and/or lacking peer reviewed results.  Which leads to the question: Is Data Scientist a marketing derived title being used to create a sense of awe around a role that is not well understood or defined.  Bear in mind, this expert would come at a high billable rate due to their highly specialized and esoteric skills.  

Is the Data Scientist to be the new silver bullet resource, positioned to solve seemingly unsolvable Big Data challenges?

If we're still looking for silver bullet solutions, I think this indicates that rigour around managing large volumes of data, data quality, database and data based management issues, data governance, metadata management have become the elephants in the room.  As such we are choosing to live with but ignore these elephants while we deal with the new distraction object called Big Data.  We can't use the old titles to deal with Big Data and not deal with these other issues so we've created this new title to obfuscate our failure to deal with our elephants.

I've derived three meaning or definitions for Data Scientist:

1) Data specialist applying the scientific method to data problems

2) A business consultant and technical data specialist and data subject matter expert

3) Data Pseudo-scientist silver bullet distracting from our elephants solution

I really hope it turns out to be 1 or 2 and not 3. 

--------

*"Basic Research is what I am doing when I don't know what I am doing"  Werner VonBraun

"It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change." Charles Darwin

It's that time of year again... winding down 2012, ramping up 2013. What changes can we expect next year?

I see a significant shift in Data Warehousing to do more and more reporting out of OLTP systems and less reliance on OLAP systems. I don't see EDWs going away quickly, but over time they will evolve into something less OLAPish and more closely tide to real time source data and systems of record.

I've always advocated not moving data from the systems of record to reporting environments, or if necessary move it as little as possible.

Data is like perishable fruit, the more you move it the less valuable it becomes. Latency causes it to spoil and become stale.

Performance characteristics on OLTP systems are now starting to allow for OLAP on the same platform - or at least closer to the system of record, and in near real time.

There's my prediction for the future.

sky diving from space

some crazy Australian rode a weather balloon into the upper stratosphere on the weekend and the from 75000 ft. jumped out. being the first person to skydive from space (or at least the outer atmosphere - as high as helium can take you) .

I'm writing this on the subway. there is a pretty young girl sitting across from me, reading "Narrative of the Life of Fredrick Douglas, an American Slave".

this stream of consciousness. this aesthetic. from skydivers to slaves. of lives lived. in all their variety . From Plato, "what is good, Phadreus, and what is not good and who need tell us these things?"

it is a beautiful day to be sentient.

The Wrong Users Manual

Does life surprize you or confuse you?   

The book, "The Black Swan," talks about events that come out of left field and surprise us.  

A great example is 911.  Two things that baffled me about 911, the coordinated successful operation and the colapse of the buildings.  I remember thinking how could someone hate that much to commit such henious act.  

The Black Swan suggests that we may have the wrong users manual on life.  When they happen we try to make sense of it in hindsight, but really we are just deluding ourselves.  If lawmakers warned us and caused us to increase security of the cockpit, we would complain about all the cost increase of flights.   Clearly no believing that such an event was possible.  

See what I mean about us not having the right user manual.

Another observation is that some disasterous IT projects that have driven companies out of business.  Clearly these decision makers have the wrong user manual.  

I commit personally to spend the rest of my life trying to find the "Right User Manual."   Remember life is a pligrimage not a destination.

Speaking Up II

Jim, a friend of mine, was a summer student hired by the Federal Government to work in Kenora to work on implementing giving First Nations people the vote.   This was an important initiative by the governement and Prime Minister Diffenbaker came to give the program his support.  Jim was one of the people in the car that was taking him to his hotel.  The hotel where he was staying had recently refused to allow a first nations person stay at the hotel.

Jim, at some personal risk, told Diffenbaker in the car about this and Diffenbaker decided to switch to another hotel.      

Many of us would have been reluctant to speak up from a very junior position when his superiors had decided where the Prime Minister would stay.  His example inspires me to stand up for my principles.  

Poetry Versus Prose

Graham Boundy and Harvey Gellman would often say that this book would have made a good article.  Recently I read a peice that suggested expressing your ideas in a poem was much more efficient.  So maybe often a book should have been a poem.

I have not completely digested the idea but poetry leaves much more to the reader and does not have to be linear.  However some the best ideas are expressed in poems.

However the reader must work on what the message means to them as opposed to the poet's message.   

This I find a fascinating idea especially for complex ideas that can said simply in a poem.

"Save time with rhyme, 

Can you get a message that is sent from the heart

I want to help without all the prose

Just follow your nose

Trusting the reader to read with heart

Means everyone has done their part"

Now wasn't that fun.

Here is what Joseph Brodsky says:

“The way to develop good taste in literature is to read poetry,” he wrote. “[It] is not only the most concise, the most condensed way of conveying the human experience; it also offers the highest possible standards for any linguistic operation — especially one on paper.

“The more one reads poetry, the less tolerant one becomes of any sort of verbosity, be that in political or philosophical discourse, be that in history, social studies or the art of fiction. Good style in prose is always hostage to the precision, speed and laconic intensity of poetic diction. A child of epitaph and epigram, conceived indeed as a shortcut to any conceivable subject matter, poetry to prose is a great disciplinarian.

“It teaches the latter not only the value of each word but also the mercurial mental patterns of the species, alternatives to linear composition, the knack of omitting the self-evident, emphasis on detail, the technique of anticlimax.”

Speaking Up

Sometimes we are annoyed when someone takes a controversial position in a discussion or meeting.   We want people to agree with our position and do not like contrary opinions.  However often if we are open to listening we can learn some important things.  Often the senior people dislike contrary opinions and dismiss them.  They do not appreciate that the most value comes from contrary opinions and not "the yes men."   

Sometimes we are so intent on pushing our own ideas we fail to see the value in other points of view.  

Often I have taken positions that are contrary to the "party line' and have found my ideas ignore.  Maybe I need to find new ways to express my ideas in more palatable.

I often think of the fable about the King's new clothes.   The little boy was ignored because he was stating the obvious, not the popular position.  

Missed Me?

Has anybody noticed I have not publish a blog since July 17th?   If you missed me and would like to see more, please let me know by a comment or a Like.

Giving People the Benefit of the Doubt

Do you dismiss opinions that do not jibe with you view of the world?  Do you consider that they may be seeing things from a different perspective?  

I find seeing the others point of view when is not alignned with mine very difficult.  

I just heard a story that one person objected to the actions of a committee of a volunteer organization.  The person accused the committee of acting contrary to the organization's constitution.  The whole issue became very devisive and resulted in most of the executive resigning.  The objector even threatened people.   I suspect the organization may not survive.  As the saying goes, "One bad apple can spoil the barrel."

Because of the person's attacks, the other people got their backs up and were unable to see the person's point of view.  I wonder if somebody had seen both sides and acted as a mediator would a resolution been found without putting the whole organization in jeopardy?

In many situations, we often dismiss the person's objections as a personal attack and the motivation as destructive.  That means trouble.

Look at the others point of view from their vantage point and maybe a new reso;ution can be found that meets everybody's objections.  

Meetings on the Phone

So many meeting these days are held on the phone rather than in person around a table.  How to manage effective meetings?  In person meeting are challenge enough and often are not done well.  Adding the complexity of a phone meeting, adds significant challenge for all participants, especially the chairman or facilitator.

I recall a great video by John Cleese called Meeting Bloody Meeting which is a fabulous training film.  See a preview on youtube Preview Meeting Bloody Meetings.  

I have noticed many people really are not participants in phone meeting and I suspect are doing other things while the meeting progresses.  If the person is only there for information and a active participant, then maybe that is OK, but I am not sure.

The chairperson who is keeping people engaged by asking peoples input is really important.  Do not assume silence means anything.  Keep a list of all participants and actively ask individual's opinions.   Also at the beginnning of the meeting, going around the table to bring everybody into the meeting by getting some input on something relatively uncontroversial.  A simple update from each person would be a good warmup  A good warmup is as key to the success of a meeting as a good agenda.  

I wonder how many people have any training on phone meetings.   We spend so much time on these things wouldn't it be smart to become more effective.  

I would like to hear any ideas people have about this subject and any training that is available.

Curiousity - A Key To Listening

Why do we all have such difficulty listening to others?   For me I often distracted by my thinking about what I am going to say next.  I am thinking "How can I share my ideas" rather than find out what the other person thinks.  

How can I make myself more curious about what the other person thinks and learn from them as opposed to try to show how smart I am.  

I am going to try some really open questions for the next week.  Let me give you some examples:

• OH?
• Tell me more?
• Interesting?
• (silence)
• MMM?
• Can you explain that?
• What's you opinion on ________?
• What do you think?

I will report how well I do and see if it improves my listening.    Thoughts?

Importance for Relating to Others

Many people have criticized business and engineering schools for not teaching any courses on how to wins friends and influence people.  Tom Peters and Dale Carnegie come to mind.

However the key success factor for people to be successful in taking a leadership role is their ability to relate and influence others not their technical capability.  

The trouble with teachung these skills is that the student must be ready and they must believe it is important.  One interesting point is that Einstein was not the greatest scientist of our time that makes us adore him.  He was more charistismatic than some of the others.

The leader of the Manahattan Project was not a great scientist but a great leader.  

I bet we all can think of the people we would follow because of the ability to lead as oppose to their technical ability.   I also believe that these skills can be learned if the student is ready.   Also when the student is ready, the teacher appears.

I think a good start would be reading and using "How To Make Friends and Influence People."   Check out a new version for the digital age. 

 

Bias For Action

When Dan Pink asked Tom Peters what was the most important principle from his book 'In Search of Excellence', he said without hesitation "A Bias for Action."  Dan Pink thens asked what was the second of the eight and after a long pause he said "A Bias for Action.'

He said he got in trouble with McKinsey because they wanted emphasis on STRATEGY.  They created great strategies for clients that they could not implement.  I have seen this over and over again.   I see it happening when ever a large consulting group is hired for to develop a strategy.  I am amazed often that the organization survive these grand strategies.

My approach was always developing plans that produced Rapid Results in the context of a strategy that set direction.   For example direction set in the Ten Commandment or the Golden Rule, as opposed to the whole Old Testement.  

How many companies have strangled themselves with the Grand Strategic Plan?  I bet you can think of several.

A company did not take my advice went with the big plan and three years later and millions of dollars later they had nothing to show for the great inventory management system that was attempted.   I felt really badly that I could not sell them on my approach but they wanted the GRAND plan, not action.

I am so pleased that I have discovered Tom Peters agrees with me.   

Career Choices - Careful Who You Talk to

When I was in high school and considering careers, I seriously considered going into medicine.  I had a really good friend whose father was a doctor so I asked if I could talk to him.  If I recall correctly one of the first things he said to me was that I would have to accept that I would not make money for ten years.  Interestingly I do not recall him talking about the work at all.

Later I discovered he was not happy practicing medicine as a family doctor in a small town.  In hindsight I should have talked to the most respected doctor in town or at least my parent's good friend who I think was happy in his job.

These days we make career changes much more frequently and in choosing our direction we must be careful who we talk to.  

in ontrast to my story, my daughter wanted to be a doctor and was invited by the medical examiner, who was the father of a friend, to watch an autopsy.   I asked how she like it and she said "It was amazing.  We finally found he had a blocked artery."   I asked if the smell bothered her and she said "It wasn't that bad."

She is now a fabulous family doctor that is loved by her patients.   In fact she almost did not get accepted into to med school but she was determined.     

Speaking of changing direction, at one stage I decided I wanted to into strategy work rather than IT work.  I teamed up with a very knowledgable strategy consultant and learned tons.  However he was more academic than a business fellow.   After looking for business in strategy work I realized that I should not turned my back on the IT clients.  I then did IT strategy work and made a very successful transition.  My lessons from that was do not turn your back on your strengths but use them in the transition.

Three stories with diffferent endings but lessons in each one.

Steps to Write Better

 Some ideas from a writing coach's blog I follow.  see http://www.publicationcoach.com/blog/

1. Always start early. One of the best ways to make writing less onerous is to begin it as quickly as possible. As soon as I have an assignment from a client or a boss, I start making some notes. Then I make a plan - allowing myself plenty of time for thinking (which must occur before writing) and even more time for editing (which is where the real improvement to writing occurs.) 

 

2. Avoid the blank page. In my view, there is nothing worse than the blank page. It's damned intimidating and it taunts us with its relentless emptiness making us fear our own minds may be equally empty.  I strive to get something - anything - written on the page as soon as I can so I can always feel as though I'm improving something rather than simply starting it.

 

3. Don't awfulize. I was a born awfulizer - always seeing the worst in every possible situation. I used to imagine my writing to be horrible and inept. Finally, I realized that this attitude was self-defeating. Writing is just talking on paper. (I now do this as fast as I can and refuse to worry about spelling or grammar in my first draft.)

 

4. Start with a mindmap instead of an outline. Mindmapping allows me to tap into the creative part of my brain, uncovering stories, metaphors and connections that I would have never encountered with an outline. If you're a subscriber to my Tuesday newsletter, you will have received a fr/ee copy of my booklet on mindmapping when you signed up. (Read it again. )

 

5. Practice daily. Writing is like exercise: The more I do, the better I get at it. I ensure my writing muscles get a consistent workout by writing for an hour a day for five days per week.

 

6. Don't edit while writing. This was a hard-fought battle for me but it revolutionized my writing. Now I produce my first drafts ever so much faster and with minimal angst. It was hard work to get to this place, but well worth the effort.

 

7. Write with a timer. When I write I have a timer that clicks in the background. I find the sound comforting and inspiring. It's also really effective at keeping me on task. When my clicker is operating I don't check email or Facebook; I feel as though I'm taking part in a race, and this is fun.

 

8. Make big goals smaller. Writing can seem overwhelming. For this reason, whenever I'm given a big job I immediately break it down into smaller tasks. Think about the assignment. Check. Do some research. Check. Prepare a mindmap. Check. Write a first draft. Check. Edit the first draft. Check. None of these tasks is big enough to freak me out so I just concentrate on collecting my check marks as quickly as possible.

 

9. Read voraciously. The best writers are indefatigable readers. I read widely and well and I get a wealth of tips from what the masters ahead of me have accomplished. (Check out my word of the week and sentence of the week postings in my blog.)

 

10. Give yourself plenty of rewards. I am slowly learning how important it is to reward myself frequently and generously. Writing is not as hard as bricklaying but it means I must concentrate and apply myself. Modest rewards - a book, a magazine, a box of a specialty tea I like - are good for modest accomplishments. Major accomplishments require something bigger. I have learned not to be stingy.

 

 

7 ways to tame your email monster

Here are some suggestions from a newsletter I read.  It has to do with dealing with email while writing a report, proposal, requirements document or memo.

1. Never try to write when you might receive ANY alert (either audio or visual) about emails lurking. Turn off your notifications or simply shut down your email software. Writing can sometimes be hard and when we’re tired or frustrated it’s all too tempting to switch to doing something more entertaining like reading an email.

2. Don’t fret about what you might miss when you’re signed off. If this thought distresses you, simply calculate a minimum acceptable response time for your employer. The vast majority of businesses will see a two-hour response as “very quick” so that means you need to check your email no more than four times per eight-hour day.

3. Don’t check your email first thing in the morning. Most people have more energy in the morning. This is the time when you should be writing, working on important projects or eating frogs. Why spend your precious high-energy time on other people’s agendas? Put your own work first!

4. Think about how your super speedy responses may be contributing to your email problem. If you generally respond to emails within five minutes then you’ll quickly become known as “rapid Rosemary” or “speedy Spencer.” In other words, people will EXPECT you to reply quickly. But if you establish a track record of responding within 12-24 hours, they’ll adapt to your habits.  (If they have something urgent, they will phone you.)

5. Batch your emails. Set specific times of day when you’re going to deal with your emails. If you’re working an eight-hour day, for example, you might designate 11 am, 1 pm, 3 pm and 4:30 pm. To really reinforce this goal, keep your email software turned off and launch it only when you need to send an email (then turn it off again right away) or during one of your designated pickup times.  

6. Play “beat the clock” when processing email.   Handle it once and quickly.

7. Tell your boss and/or clients what you’re doing. Be sure to tell everyone about your new policy for email. If you want extra Brownie points, ask your boss or clients to call or text you if there’s anything super urgent. And if you happen to mention that you’re doing this to improve your efficiency for them, my hunch is they’ll be both delighted and flattered.

For complete item see http://www.publicationcoach.com/7-ways-to-tame-your-email-monster/

What do you do to tame your email monster?

Questions You Wished You Asked Your Grandparents

As I got older, I have become more interested in my genealogy.   However I have learned that the dates are less important than the family stories.  I really did not learn many from my grandparents but wish I had asked more.

A friend of mine had the same idea and asked a bunch of us to what questions we wished we had asked our grandparents.  His idea would be to compose a list and then write some answer for his grandkids.  

The list is quite extensive so I will not repeat it here, but if you are interested I could send it to you.

• What are your fondest memories?
• What are your worst memories?
• What did you do most evenings when you were young?  - middle-aged?
• What were your most favorite places you visited/travelled?
• Who were your favorite entertainers?
• What were your hobbies?  - interests?  - sports?
• What were your parents like?  - grandparents?  - siblings?
• What were you proud of?
• What were your health problems as you got older?
• What special gifts do you remember getting?
• What were your hopes and dreams when you were a young person?
• What responsibilities did you have in your family?
• What particular role do you think you played within your family?
• What was a typical school day like for you?
• What were your siblings like?  - did you have a favorite?  - why?
• What factors were driving yours/your parents’ need/desire to emigrate?
• Where did you/your parents come from?
• Who did you/they leave behind? 
• What difficulties did you/they encounter?

I must get on with answering these questions.

Get Anyone To Do Anything

From a book called "Instant Influence" I extracted the following six questions:

1. Why might you change?  (for yourself why might I change?)

2. How ready are you to change  -- on a scale of 1 to 10, where 1 means "not ready at all" and 10 meands "totally ready."

3. Why didn't you pick a lower number?  If one was picked, ask question 2 for a smaller change or ask "what would it take to for 1 to turn to 2.

4. Imagine you've changed.  What would the positive outcome be?

5. Why are those outcomes important to you?  

6. What's the next step, if any?

The key is tapping into the person's readiness.  Also they come up with the ideas, not you.  This leads to action.

I remember when I first learned about readiness and learning people only do what they are ready to do.  I then decided on my next proposal to take that into account.  I had no idea how to proceed.  

Clearly the questions above point out the other person must indicate their readiness.

Another time I had a great idea for a client for a strategic planning process and completely misread their readiness.  

Tuning in to the other person's readiness is a great skill and one that we need continually work on.  

Adler. a famous psychologist, said he could detect the client issue on the first visit and it could take many sesssions for the client to discover the issue.   What patience!  If only, I could shut up and listen. 

The whole key to the steps above is listening, listening and listening.