The Perfect Problem

Problems are a fact of life, certainly if you live in the perfectly bit-oriented world of computing where all those bits must line up in their proper place, billions of them, to be processed at rates exceeding a billion times per second.  Seems on the surface to be an impossible task, but that's the comforting life for IT professionals, who relish in keeping the computing universe in alignment twenty-four hours a day, seven days a week.  And when the inevitable problem occurs, it gets diagnosed and repaired at dizzying speed.  Then it's back to programming more billions of, hopefully but rarely, perfect bits.  

But the above is only a fraction of the whole story, and in many cases is the shortest portion, at least from the viewpoint of the user whose service was interrupted for hours or days.  But this fraction is also typically the focus of the root cause analysis, which is focused on avoiding the problem in the future.  And hence a larger opportunity to improve service is wasted.  To get at the bigger picture, begin using the concept I call "The Perfect Problem".

Was is a Perfect Problem?  This will vary somewhat depending on your service agreements, but the general idea surrounds all the other "stuff" surrounding the outside of the technical problem.  Simply put, a Perfect Perfect is reported, dispatched, escalated, recovered and communicated in the way you designed.  It's all the operational aspects that can consume 90-99% of the actual time a service was unavailable.  It's all about asking questions, and a lot of questions, and getting straight answers so everyone gets better.  Some examples of the type of questions you need to ask include:

• Reported - Did the automated monitoring tools or computer operators see the problem before users started calling the help desk?  Could they be improved to shave minutes or possibly hours from the overall duration of the outage?
• Dispatched - Did the ticket get assigned to the correct group?  Where the right people paged?  Was the ticket picked up within the specified time frame?  Get the ticket get bounced back and forth trying to find a home?  Was all the information gathered to that point included to avoid wasting time asking for it a second or third time?
• Escalated - Per procedure, was the problem escalated to a problem coordinator within the proper time period?  Was management made aware of high severity problems?  Were tickets opened with vendors escalated properly?
• Recovered - After the service was fixed, did it come back up as quickly as usual, or were other steps needed?  Was all the operational start-up documentation accurate?  Could improvements to the documentation make it more clear and less error prone?  Are there ways to speed up restarting the application?  How did users know the service was available again?
• Communications - Was a message recorded for the automated help line letting users know that you know about the problem?  Did that message get updated on scheduled?  Did it include all the information that should have been there?  Were key business groups made aware of the right problems?  Was the CIO notified that the critical systems she or he cares the most about?  

In other words, a Perfect Problem went exactly like you planned it to go.  Nothing more, nothing less.

To capture improvements to make future problems flow properly, each identified deficiency needs to be generate an appropriate improvement task with a clear description and owner.  Each task remains open, monitored, prioritized and managed until resolved, and open items regularly reviewed by management.  

In my experience the longest outages are not typically caused by unusually difficult technical problems, but deficiencies in executing the surrounding processes.  And that's totally within our control to improve upon.

N-1-1

We've all heard of 9-1-1, the single number to reach emergency services which began way back in 1968.  But did you know that other countries use 1-1-2, 9-9-9 and a host of other numbers to reach various services.  For example, Brazil uses 1-9-0 to reach the police, 1-9-2 to contact medical services and 1-9-4 to find the fire department, and a handful of other sequences for more specialized services.  That would be too much for me to remember, particularly in an emergency situation.

You also likely familiar with 4-1-1, the short-cut to reach directory services, which can add a significant amount to your monthly phone bill if you're particularly lazy or forgetful.  But did you know that free alternatives exist?  Jingle Network's 1-800-FREE-411, Microsoft's 1-800-BING-411 and Verizon's 1-800-THE-INFO are provided free of charge, although some are advertising supported.  On the plus side, driving directions, sports, weather and other features may be available.  You won't talk to a human being, but with the recent advances in speech recognition, the computer is most likely going to get your request correct more often than not.

Now to the services you may not be familiar with, mostly depending on what part of the country you live.  

2-1-1 is reserved for community services such as affordable housing, homelessness, drug and alcohol programs and suicide prevention.  Many of these 2-1-1 services are run by a local United Way agency, as they are in the Dayton, Ohio area.   They also have a web site and a toll-free number to reach agencies when outside the local 2-1-1 dialling area.  More information on their service, HelpLink, is at www.dayton-unitedway.org/help.php.

3-1-1 is the non-emergency version of 9-1-1, but is available in only a couple dozen, large metropolitan areas.  Its purpose is to easily connect residents to city services and information.  Columbus is the only city in Ohio currently with 3-1-1 service, including their web site at 311.columbus.gov.  Examples of the many services offered include requesting a bulk trash pickup, reporting issues with street lights or pot holes, reporting an abandoned car or complaining about a barking dog.

5-1-1 gets you connected to traffic information and covers a large percentage of the United States, and a coverage map can be found at www.fhwa.dot.gov/trafficinfo/511.htm.  The only service available in Ohio serves the Cincinnati/Northern Kentucky region with an effort called ARTISMIS, which stands for "The Advanced Regional Traffic Interactive Management & Information System".  More information on the services offered are located at www.artimis.org.  You can also try out the service for yourself by dialling 1-513-333-3333.

7-1-1 is the Telephone Relay Service and is provided nationwide for the Deaf and Hard of Hearing with more details provided at 711service.com.  

6-1-1 is used to contact your phone provider.  The informational web site, www.dial611.com, provides more information.  Most locations should have this service, and it should be free

8-1-1 is the nationwide "call before you dig" number, which seeks to protect our underground infrastructure.  Its web site is www.call811.com.

A summary chart of all N-1-1 numbers is available at www.nanpa.com/number_resource_info/n11_codes.html.

Web Debugging with Fiddler

In a previous blog I mentioned the free Wireshark utility, which has been my number one debugging tool for several years.  Being able to see everything coming into and out of a PC, and having enough network background to glean the important details, has served well.  During a recent problem I needed to debug a web application that only ran using encrypted (https/ssl) communications.  Wireshark was able to show me what was happening with session setups and encryption exchanges, but all the application data was just a garble of meaningless characters.  Enter Fiddler, a free tool provided by Microsoft.  

Fiddler is a web debugging proxy that captures all http, and optionally https, data from any web application that can point to your loopback address (127.0.0.1) on port 8888 (the default).  The interface displays a list of request/response pairs and its status code on the left side of the screen and a detailed breakdown of the request and its response on the right.  There are multiple views available to visualize each request and response.  I find the Statistics view useful to see the number of bytes sent and received and its response time, and the Inspectors (Raw View) to see the gory details.  Other frequently accessed views are the Inspectors (ImageView), Filters and Timeline.  Third-party developers add more features to Fiddler and I've installed neXpert to generate detail reports with suggestions for improvements, Watcher to detect potential security issues and JavaScript Formatter to make JavaScript easier to read. 

Launching Fiddler can be via its shortcut or from Internet Explorer using the Tools ... Fiddler2 option.  Fiddler will automatically change Internet Explorer's proxy settings, which also affects all other applications, like the Google Chrome browser, that use the same settings.  Mozilla's Firefox, which does not use IE's proxy, can be controlled via an option installed in the lower right-hand corner. 

Capturing and displaying encrypted data requires changing some default options, located under Tools ... Fiddler Options ... HTTPS tab.   I suggest you read the information found at the "Learn more about HTTPS Traffic decryption and certificate errors" link.

Fiddler has a number of features, more for the professional developers and tester, that go way beyond simply displaying data.   You can set breakpoints and even fiddle (hence the name) with the data and inject your own.  But just seeing what's really going on "behind-the-scenes" can be eye-opening.

For more details, instructional videos and download, visit www.fiddler.com.

The Emotion of Technology Change

I began my career in Information Technology back at Wright State University in 1974.  I didn't know anything about computers when I started, but a friend that was at WSU at the time showed me some programs and I was attracted by the logic, their math-like quality and the promise of avoiding a bunch of annoying liberal arts classes.  But little in my college experiences prepared me for how much emotions play in the technology field, either good or bad.  I've collected a few of those experiences in this article to share with you.  And if you're ever needing an ice-breaker when in the land of techies, just ask then about their most hair-raising experience.  You're sure to hear an earful and get the conversation going.  

I'll start with the worst feeling I've ever had.  I was new with a company back in 1980 and was working hard at getting something or another to work.  I was sitting at the master console of the company mainframe and realized I needed to stop any new work from starting in the system.  My brain instantly stormed two possibilities.  I could "purge the initiators" or I could "hold the queue".  Unfortunately my fingers interpreted an unfortunate combination of the two and decided to "purge the queue".  Before I thought twice my fingers had entered the requisite command and the system began to get rid of everything waiting to be run or the output of everything already complete.  When my brain figured this out a few microseconds too late, it was beyond stopping.  My stomach fell like a lead weight and my hands wanted to reach inside the console and take it back.  It was all I could to avoid vomiting on the computer room floor.  Fortunately I worked with some experienced, and nice, folks who had "been there and done that" and cut me a break.  I just wanted to hide.

On the other side of the emotional teeter-totter, one of most positively exhilarating experiences was back in the early days of the Internet.  Like most of us I  rode the modem speed curve as fast as I could.  Loved 2400 bps (bits per second), died and went to heaven when 9600 bps became reality and dreamed of the day that the promised 56,000 bps would actually work on my home PC.  The intellect could only imagine what a really high-speed connection would be like.  My time came during a trip to the IBM Raleigh North Carolina briefing center, where I stayed at the Washington Duke hotel.  I had heard they had a T-1 (1.544 megabits per second) to the Internet and couldn't wait to plug in and take off.  Yes, it was fast.  Yes, it was awesome.  But the surprising emotion was a sudden realization of the potential of the Internet.  Anything, anywhere, and in an instant.  It was as if, in my mind, the Berlin Wall had fallen that day.  I sat in my hotel room, my world changed, forever.

Some of the best emotions are deep-rooted in our past.  In 2009, one-quarter of Americans do not have a landline for telephone service and that number is increasing very day.  But I grew up with only a landline.  To date myself, when I was growing up my family had a six-digit phone number.  Crestview-3-8-3-2, which translated to 2-7-3-8-3-2 (the "C' and "r" in Crestview translating to "2-7").  And we had heavy rotary phones that rang like church bells and all the kids (five of us) would run to answer it when it rang.  And you actually had to walk over to where the phone was to use it.  Imagine that.  With that as a background we speed ahead to last year, 2009, when my wife and I, after moving from a "real landline" to a cable-provided "fake landline" the year before, decided to "cut the cord", eliminate the cordless phone and go with our cell phones and Skype.  In our heads, not a big deal.  In the emotions of our heart, a totally different deal.  A solemn call to Time-Warner.  The removal of the phones from each level of the house.  The feeling of being cut-off from the world.  Took awhile to feel whole again.  And we do miss the loud ringing on occasion.

  

Finally, I bought my first app-phone late last year, a Motorola Droid.  I've had an iPod Touch for a couple years, like the apps, but always feeling Wi-Fi needy.  I've had a love-fest with the Motorola RAZR since they first came out and have owned three.  But the Droid, again very unexpectedly, came with a stronger emotional response.  I was connected.  Always.  I could look it up.  Anywhere.  I took my stuff with me.  Anything at all.  Two email accounts, two calendars, music, videos, Twitter, Facebook, chats and messages.  Maps, alarm clock and directions, oh my!  Excited, sure, it was a dream come true.  But the unexpected emotion that swelled up was that of being overwhelmed.  Too much.  Too often.  Out of control.  Maybe some "dial tone therapy" would help. 

The Outcome-Value Statement

This article describes a concise and effective method to communicate a project, a requirement or even an organization's purpose to multiple audiences, each listening for their part of the message.  It does this by linking the work being done to the value being delivered through its expected outcome or outcomes.    

The work being done can be the list of various projects or the major components of projects.  It can a list of requirements for a new service or a new set of rules being considered.  This is the place to begin creating the Outcome-Value statement.  Simply write them down in a list with the most significant items first.  While the list could be very lengthy, it's best to summarize enough to keep its size to ten or less.  

The next step is to write down the Values you expect to gain. These fall into three categories: cost, service and risk.   It's not time yet to link the value to the work being done.  Just list the values for the entire effort.  It's okay to make these somewhat fuzzy for the purpose of this effort.  It's not intended to replace a business case analysis.  So a statement of "reduce maintenance costs" or "improve service reliability" is sufficient. 

The third step is usually the most difficult, although it comes faster as you become familiar with the process.  This involves stating or predicting one or more of the resulting Outcomes.  An Outcome is simply how are things different.  How does a business process change?  How might the people using a new service view the difference?  Perhaps a current service is being eliminated as part of this project.  The Outcomes will typically show you where your change management issues exist or where communication to affected parties will be required.  Most importantly an Outcome is not a cost, service or risk statement.  Saving a million dollars is a Value, not an Outcome.  "Reducing product recalls" is an Outcome, not a Value.  Outcomes may not be all that exciting, for example, changing a supplier that results in a million dollar savings.  It's an exciting Value, but not an exciting Outcome from your point of view.

The final step is to assign each Value to one or more Outcomes and link each work item to one or more Outcomes.  The result should clearly show what is being done, how your world will change and why it's a good thing.

Let's use implementing an email retention policy as an example.  Prior to this project people could freely keep or delete anything in their email mailbox, but new regulations and service disruptions are requiring a change.  Let's start by writing down a fictitious, but realistic, set of proposed rules. 


     - Emails older than 90 days will be automatically deleted  
     - Each user's mailbox is capped at 250 megabytes of storage
     - Email backups tapes will be erased after 30 days
     - Users can only move emails requiring longer-term storage to the content management service
     - Automatic deletions and erasures will be halted as required by legal proceedings

Step two is listing the Values expected.  

     - Reduce disk and tape storage costs (cost)  
     - Improved email system performance (service)  
     - Fewer "smoking gun" emails being kept (risk) 
     - Compliance with court orders (service) 
     - Retaining business records and knowledge (service)

Now the hardest part, the Outcomes.  Even in this made-up example it took about ten minutes to clearly state what is different.  Looking at it from the email user's viewpoint helps see the Outcomes.

     1. Email Becomes a Transitory Tool 
     2. New Processes for Retaining Emails for Legal and Business Requirements

Putting this all together and linking the new set of rules to the Outcomes, the final Outcome-Value statement is produced.  Each Value is listed as a bulleted item after its associated Outcome and after each new rule the Outcome it supports is listed in parenthesis.  

Email Retention Outcome-Value Statement

1. Email Becomes a Transitory Tool

     > Reduce disk and tape storage costs  
     > Improved email system performance 
     > Fewer "smoking gun" emails being kept

2. New Processes for Retaining Emails for Legal and Business Requirements

     > Compliance with court orders  
     > Retaining business records and knowledge

     - Emails older than 90 days will be automatically deleted (1)
     - Each user's mailbox is capped at 250 megabytes of storage (1)
     - Email backups tapes will be erased after 30 days (1)
     - Users can only move emails requiring longer-term storage to the content management service (2)
     - Automatic deletions and erasures will be halted as required by legal proceedings (2)

My Web Site List

You might think from my earlier blogs that I don't spend all that much time on the Internet.  Nothing could be further from the truth.  I write this blog on Google Docs, so I can work on it wherever and whenever.  I have a Google Chrome browser up at all times with seven Google services always open.  I have Firefox up at all times to handle RSS feeds and Google searches.  I regularly seek information from the three sources of all knowledge (Google, Wikipedia and YouTube).  Take my browsers away and I stumble upon this Earth.  

We all have our favorite web sites and I'll share mine with you.  Most you're probably familiar with, but even one new winner is worth perusing the list.  Here goes....

• Kayak.com - I use this for all my travel planning.  It is by far the best web site I've found for quickly finding good airfares.  It works like this:  (1) pick your cities and pick your date, (2) all the data for all the flights is sent to your browser, and (3) by using check boxes, sliders and buttons you can narrow or widen your search criteria and never go back to the web site for another long search.  
• SportingNewsToday.com - Part of my morning ritual.  Make coffee, let the dog out and then read "my morning paper".  SportingNewsToday has the best layout.  Quick and easy to find just the parts you like or skim the day's issue.  More like reading "the sports page" than drilling up and down the typical hierarchical web site.
• Gutenberg.org and Librivox.org - Books in the public domain are available at Gutenberg.org.  Audiobooks in the public domain are at Librivox.org.  I particularly like Audiobooks for car commutes and have gone through most of the Sherlock Holmes and Wizard of Oz books (there are over a dozen of each).  Volunteers record the Audiobooks, but I think you'll be amazed at how well most of them do.  
• Pandora.com - There are a number of good music sites and Pandora is my favorite, although Grooveshark, Finetune and Last.fm are among the other fine choices.  You build "stations" in Pandora based on an artist or song, and it uses that to play songs by the artist and songs similar to that style.  You can rate individual songs "thumbs-up" or "thumbs-down" to further tune the station to your liking.  Even cooler running on my Droid phone in the car.  "Radio" with no commercials. Sweet.
• Cio.com - My personal choice for balanced and professional reporting of technology news and topics.  Digg, Slashdot and Lifehacker are good too, but a little too much chaff, opinion and irrelevance.
• GeekBrief.tv - Cali Lewis delivers up-to-date technology news via a 3-5 minute video several times a week.  Particularly fond of Apple computers and Japanese robots, Cali delivers a lot of information in a very short time with shots of humor.  Nice not to read for a few minutes every morning, but it's still part of "my morning paper".
• Shoeknots.com - I just had to mention this web site although it contains a single page.  One day, many years ago, I was ranting about my new pair of dress shoes.  The dang shoe strings would not stay tied.  Now this was a problem I've had all my life, but this pair of strings was really bad.  I went on the Internet to find someone who sold "slipless" shoestrings.  I would pay anything.  I found Shoeknots.com, which taught me how to properly tie shoelaces.  I had been tying "granny knots" all my life.  Now I tie "square knots".  This web site taught me to seek out the knowledge of others.  A defining moment as I truly entered the age of search.  
• Bloom County and Dilbert - At www.gocomics.com/bloomcounty and www.dilbert.com.  Gotta have my morning dose of humor.  Bloom County is still hilarious the second time around.