Random Thoughts from a CTO: Software Engineering

Do you know what a CTO does?

Thu, 01 Jun 2006 15:03:06 GMT

CTO stands for Chief Technology Officer (in some companies, also referred to as Chief Technical Officer). They are usually part of the company's top management team. They report to the CEO, President or COO.

Here is a rundown of my duties in this position:

KEY RESPONSIBILITIES:

Define, develop and deploy an Information Technology (IT) architecture that maximizes revenue, minimizes costs, helps get and keep IT aligned with business goals and provides a firm, flexible foundation for the future.
Ensure that all product releases are properly implemented, integrated and supported.
Become the corporate subject matter expert on all trends and developments in Software Development that are relevant to the business.
Oversee compliance of product development resources with relevant standards and appropriate best practices.
Keep other team members and senior managers aware of the value of the IT architecture and implications of proposed changes or additions.
Help educate others outside of product development on the value of the IT architecture to the business and their roles within it.
Provide the strategic technology vision of enterprise-wide technical architectures and direction for the company. Work with others in product development to implement that vision.
Build/enhance/motivate/manage the product development department.
Executive management of the company's IT presence, including full responsibility and accountability which is exercised through active membership in the Executive Committee.
Coordination of product development budgets, staff plans, or performance reviews with project managers and supervisors.
Hardware and software forecasting, purchasing, or contracts.
Recruitment and hiring-- either of permanent or contract staff.
Accountability for all aspects of on-time delivery.

INTEGRATION WITH ORGANIZATION:

Work with Executive Committee to create a workable game plan for company's integration (short-term and long-term) using current and future technologies.
Manage cross-group integration projects to ensure quality execution of this game plan.
Build knowledge sharing within the product development department and work to develop appropriate standards for architects, tools, etc.
Seek opportunities to coordinate/share/re-use code or technologies, ensuring that we don't "re-invent the wheel" in multiple groups.

SECONDARY RESPONSIBILITIES:

Evaluate the technical underpinnings of other businesses (partners, acquisitions, competitors, etc.) as requested by company's senior managers.
Investigate new technologies and advise on their relevance to company.
Educate company staff on technology issues.
Build links to other outside technology groups and consultants to help build both expertise and access to external resources.

A difficult definition of simplicity

Wed, 31 May 2006 15:31:47 GMT

Brad Appleton over at his ACME blog has a great post today called "Simple ain't Easy: Myths and Misunderstandings about Simplicity". No matter what kind of product you are creating, the goal is always to make it simple. But what does that mean exactly, and how to you get there? This is what Brad was struggling with and did extensive research to get to the bottom of simplicity. What did he discover?

"After mulling-over all of those, I think it's fair to say that while "Simplicity" may be, well, "simple", truly understanding "simplicity" is in fact quite hard!

Simplicity involves being able to see the whole from a system's thinking perspective while at the same time being able to focus in on what is relevant and essential and how it impacts the rest of the system.
Sustainable simplicity often has to evolve or emerge on it's own from a set of simple guiding rules.
The opposite of simplicity is complexity (as opposed to "hard" or "difficult" or "time-consuming" or "labor-intensive")
In mathematics, simplicity is often "elegance" and is more than just the intersection of "what is necessary" and "what is sufficient"
In architecture, "simplicity" is often synonymous with "beauty"
Hiding complexity isn't the same as removing complexity.
Many of the tools we use to manage complexity in system's design may in fact add more/new objects to hide complexity or separate concerns
Minimizing dependencies throughout a system is more critical to simplicity than minimizing the number/types of objects
Occam's Razor does not in fact say that the "simpler explanation is better" ... it says that the explanation that makes the fewest assumptions and poses the fewest hypotheticals (i.e., minimizes the number of "given" and "unproven" conditions) is the most preferable because it is easier to comprehensively test in order to prove/disprove."

Read more in his post with plenty of links to other resources he found in his research.

Putting the puzzle together

Mon, 27 Mar 2006 18:31:24 GMT

Over the last couple of weeks, I did something that I haven't done in a very long time. Putting together a 2000-piece jigsaw puzzle. It took me 2 weeks but I finally got it done. As I was working on the puzzle, I found that my strategy on getting it done changed over time.

Usually when I work on a puzzle, I do what many do and find all of the "edge" pieces. Why? Well, the thought is that if you complete the "frame" of the puzzle, it becomes much easier to work on the insides of the picture. However, I quickly found that there were too many pieces to go through to find all of the edge pieces. So, this strategy didn't work for this situation. Next, I tried to separate pieces into colors. This strategy was a little bit more effective since my puzzle had a lot of red pieces (it was a Coca-cola themed puzzle), so I was able to find many of these pretty quickly. Still too many pieces at the beginning to go through, so this strategy wasn't enough to get going on the puzzle. My next strategy is to find recognizable pieces, whether it be lettering or something that is recognizable and put those pieces together to form sections of the puzzle. I found this strategy to work very well for this particular puzzle as there were plenty of words ("Coke" or "Coca-Cola"), and art (faces of people) to find. I start getting part of the puzzle together. However, at a particular point I found that this strategy wouldn't work on its own since the pieces weren't as distinguishable after a while.

So, how did I get the puzzle done? It took a combination of these strategies to achieve my goals. Not one particular one would get me there. So, I would take a section of the puzzle, find colors, edges, or recognizable pieces and work on a particular "theme". Once I got stuck on that one or got the theme to an acceptable amount of completion, I would take another part of the puzzle and go through that section. I continued down the path until I had most major sections completed and could go back and "fill in the blanks".

This is why Agile development can be so effective. Just like I was overwhelmed at first with the number of pieces, project teams can quickly get overwhelmed on a project that can take several months or years. There is just too much to think about and you don't know where to start. You can then try to define the high-level specifications or architecture, similiar to finding the "edge" pieces but sometimes those are hard to find overall for a large project and become overwhelming. This is where traditional software development falls apart, in a project where the scope is large and roughly unknown. You know the goal but you can't figure out how to get there while making progress along the way that is noticable. For the first few days, it didn't look like I was getting anywhere on my puzzle until I thought about doing things in chunks or themes. By doing the work in themes, I was able to take certain strategies (requirements, design, implementation and testing) that do work for any kind of project and make them more effective my focusing on a piece of the work. When I had the majority of the theme completed (at my acceptance level), I moved on to the next theme. At the end of the project, I can complete the project by doing those tasks to "fill in the blanks". By doing work using Agile, teams can show more progress on the big picture by demonstrated working areas of the project over time.

Would it still have taken me two week to complete the puzzle? It's hard to say, but chances are that I might have given up on the puzzle if I didn't feel that things were progressing well enough. Same goes with projects. If you can't show progress, project stakeholders may give up on your project and cancel it causing a lot of wasted time, effort and money. However, if you are able to show promise and be able to see pieces coming together, you not only encourage your stakeholders but the project team gains momentum and motivation in getting it done. I found my pace accelerating as I was getting through the puzzle eager to see the puzzle in its completion. Agile teams would call this "velocity", and studies have shown that doing work in smaller iterations that teams actually get more work done consistently through the project because they are motivated by their own progress.

Am I saying that software development or agile projects are similiar to putting together jigsaw puzzles? Only in the areas I mentioned above. Where they differ a lot is in their level of predictability. With a jigsaw puzzle, there is a definiite goal that is predictable. I know the number of pieces and they won't change. The picture will be the same no matter how many times I put together the puzzle. With software, things are very unpredictable. Even with the best planning, I have yet to see a project that delivered EXACTLY what it promised or planned to deliver. Customers change their minds, their needs change, technology is changing, there are a lot of moving parts. By doing things in small iterations, you can focus on what is of value now (or at least in the next several weeks). Beyond that, anything is up for grabs. Each iteration you look at things from a new perspective - "what is needed next from either a customer's value perspective or technical perspective and do that next". Agile handles change much easier. Why? Well, in traditional software development the scope is always looked to be "fixed", while in agile it is the cost and schedule that are fixed but the scope that can be altered. Cost and schedule can be very predictable (you have this many hours available with these resources), whereas scope is very unpredictable. The team will be much more successful if they can implement pieces that are predictable while researching the unpredictable pieces as they are needed (also called "spikes").

Can't wait to work on that next project...um, I mean puzzle!

Business Driven Development

Mon, 24 Oct 2005 19:16:40 GMT

IBM Rational has come out with six key priniciples based around what they call "Business-driven development". Many of these principles seem to be born out of agile or iterative development best practices, but I really like how they have bundled these together. Also, for each they provide values, patterns (repeatable processes), and anti-patterns (processes to avoid). Here is a summary of each, read more in the article:

Principle #1: Adapt the Process

Benefits: Lifecycle efficiency, open/honest communication of risks
Patterns: Precision and formality evolve from light to heavy over the project lifecycle as uncertainties are resolved. Adapt the process to the size and distribution of the project team, to the complexity of the application, and to the need for compliance. Continuously improve your process.
Anti-patterns: Precise plans/estimates, track against static plan management style. More process is better. Always use the same degree of process throughout the lifecycle.

Principle #2: Balance competing stakeholder priorities

Benefits: Align applications with business and user needs, reduce custom development, and optimize business value
Patterns: Define and understand business processes and user needs; prioritize projects and requirements and couple needs with software capabilities; understand what assets you can leverage; and balance user needs and reuse of assets.
Anti-patterns: Achieve precise and thorough requirements before any project work begins. Requirements focus the drive toward a custom solution.

Principle #3: Collaborate across teams

Benefits: Team productivity, better coupling between business needs, and the development and operations of software systems.
Patterns: Motivate people to perform at their best. Collaborate cross-functionally across analysts, developers, and testers. Manage evolving artifacts and tasks to enhance collaboration and progress/quality insight with integrated environments. Ensure that business, development, and operations teams work effectively as an integrated whole.
Anti-patterns: Nurture heroic individuals and arm them with power tools.

Principle #4: Demonstrate value iteratively

Benefits: Early risk reduction, higher predictability, trust among stakeholders
Patterns: Adaptive management using an iterative process. Attack major technical, business, and programmatic risks first. Enable feedback by delivering incremental user value in each iteration.
Anti-patterns: Plan the whole lifecycle in detail, track variances against plan. Detailed plans are better plans. Assess status by reviewing specifications.

Principle #5: Elevate the level of abstraction

Benefits: Productivity, reduced complexity
Patterns: Reuse existing assets, reduce the amount of human-generated stuff through higher-level tools and languages, and architect for resilience, quality, understandability, and complexity control.
Anti-patterns: Go directly from vague high-level requirements to custom-crafted code.

Principle #6: Focus continuously on quality

Benefits: Higher quality and earlier progress/quality insight
Patterns: Team responsibility for end product. Testing becomes a high priority given continuous integration of demonstrable capabilities. Incrementally build test automation.
Anti-patterns: Postpone integration testing until all code has been completed and unit-tested. Peer-review all artifacts, rather than also driving partial implementation and testing, to discover issues.

- via BizzBangBizz

It can't be done...or can it?

Thu, 20 Oct 2005 15:28:54 GMT

Pretend you are taking a week long trip with a friend. You start packing and end up with a couple of large suitcases. Your friend arrives with one medium suitcase. You start thinking, "What are they thinking? They don't have enough clothes to last the week." You arrive to your destination. When you start unpacking, your friend's clothes are all nicely folded and no wrinkles. Your clothes look like a mess and will need to be ironed. Your friend is able to put things into drawers quickly as well as the bathroom items. You have to go through your bag and it takes you longer to find things and put them in their place. However, your friend has less clothes than you. As the week progresses, you begin to see that your friend has found ways to reuse what they wear - different shirts with the same pants - yet still appear like they aren't worn. You on the other hand, find that at the end of the trip you had clothes that you didn't wear, but had to carry around. Now you start thinking, "My friend has a better way. I've got to learn from them for the next trip!"

Was the friend always this organized? Probably not, they probably learned by somebody else. Experienced travelers have tricks on how to properly fold clothes and how to maximize every spaces in a suitcase. They have also learned what's "good enough" and not to take more than they need. Even if they end up needing fresh clothes, they will end up using a local cleaners or wash their own clothes instead of taking too much. They have learned how to best put clothes and other accessories together to maximize their use.

As my development staff is asked to reduce cycle times through iterative development using Agile best practices, their initial reaction is "That sounds all good, but we have found things to take weeks or months and that won't work with this approach." They are correct, it won't. But, just because you have done things a certain way doesn't mean that it is the ONLY way to do things. We have learned how to do things as individuals. We will now need to do those things together as a team. We have learned that each person plays a particular role and has their own specialization. We will now need to remove those specializations and have people wear multiple hats. We have learned to think about everything up front is much detail. We will now need to learn to do just enough for now and think about details when it is appropriate later. We have learned to break the work and requirements down just enough to make sense for a 6-8 month project, we will now need to refine that work to make sense for 2-4 week iterations.

What's great about agile is that it isn't just "theory" anymore, people have found it to work and improve the flexibility and responsiveness of the team to provide working solutions to customers. We can tap into these "experienced travelers" and figure out how they have done it a different and better way. It just seems strange to us because we have learned how to do things differently, not necessarily better.

A Lean Paradigm Shift

Fri, 07 Oct 2005 20:34:02 GMT

Lean can mean different things to different people.

For some, lean can refer to how you maintain your physical shape through dieting and exercise. For those that have been successful in staying lean, they will say that it takes a paradigm shift of how you take care of your body through what you eat and how you exercise. When you start the process of becoming lean, you begin to cut out those things that are unhealthy and make you fat. You begin to eat better. You begin to feel better. You start seeing results in the mirror. You also are constantly monitoring your weight, heartbeat, blood pressure, caolries, etc through the process. For some of us, myself included, we get lazy once we reach this point and start falling back to bad habits -- at first, a mistake here and there on what we eat, or missing a workout or two. Then, you stop getting on the scale and recording your results. Then, before you know it you are not lean again!

For others, lean can refer to how you maintain your financial status through budgeting and tracking your expenses. For those that have been successful in saving money and making more through investings, they will say that it takes a paradigm shift of what you do with your money and where it goes. When you start the process of saving money, you begin to determine where your expenses are going and which of those expenses you can eliminate that takes you from your goals. You also look at ways to increase your income with those savings through investments. You also look to remove your debt. You begin to save money. You begin to feel better. You start seeing results in your portfolio. You also are constantly monitoring your income flow, expenses and net worth through the process. For some of us, in this case myself not included, we get lazy once we reach this point and start falling back to bad habits of spending excess money, getting into debt, and eventually not saving money. Then, before you know it you are not financially set again!

In the software engineering industry, lean is now referring to your cycle time - how quickly you get to a working solution for the end customer without losing quality. For those that have been successful with this approach, they will say that it takes a paradigm shift of how you development and manage your software solutions. When you start the process of reducing cycle times, you begin to determine where communication and collaboration need to happen, what activities or processes take away from reducing cycle times, how we do our jobs and the way we interact with other areas. You will begin to see results. You wil see the customer is more satisfied. You will find better ways to get your work done. You also are constantly monitoring your estimates, work completed, costs, quality much better. For some of us, myself a little bit, it is very easy to fall back into old habits of traditional and formal processes because you are so familiar with them and may get discouraged early on in not seeing immediate results you are hoping. You then begin to see things slip, take longer and less of a feeling of accomplishment either by the team or the end customer. Then, before you know it you are feeling that things aren't getting done as they should!

There are many other scenarios that this lean approach can take - time management, stress management, strategy, project management, the list goes on and on. What is getting in the way of your goals? How do you improve on those goals? What do you do to maintain those goals? How do you measure the process of those goals? Once you figure those things out, you need to continue the discipline to maintain them. Focus on the goals and don't get lazy!

Agile Manifesto Explored

Thu, 06 Oct 2005 19:31:48 GMT

The Agile Manifesto is a document that was produced by members of the Agile Alliance as a common mission statement for people that are doing agile software development. Here is what the manifesto says:

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Let's discuss each part of the manifesto in more detail:

Individuals and interactions over processes and tools - Processes and tools should serve only one purpose: to help the individuals involved be able to do their jobs better. If the processes and tools become too complex, people start having to manage the process instead of their work. They also start to use processes and tools instead of talking to each others (for example, email or a corporate intranet). You begin to lose the context of the message. You also lose the opportunity to dialog with others that will not only streamline the decisions but probably make better decisions than on your own. Processes and tools are necessary but not at the expense of the value of individuals and interactions.

Working software over comprehensive documentation - Too much emphasis in the past has been made regarding using formal documentation to replace communication as a way to hand off between members of a project team. In my past, I have seen consultants come into an organization with their sole responsibility to produce volumes of documentation and then leave. Months of effort went into this effort, along with lots of money spent. I was part of the team assigned to do the implementation. Not only did the documentation become outdated as we went through the implementation and needed to change some of the requirements, but they were very difficult to figure out. With the people gone that did the work, we ended up discarding the documentation. Isn't the purpose to get to working software and not to make sure that your documentation is meeting standards? Determine the minimal level of documentation is needed to ensure that the details of conversations are captured, but don't provide documentation to REPLACE those conversations! Documentation is important but not at the expense of getting to software that works.

Customer collaboration over contract negotiation - I have been in meetings with project stakeholders that have feel like you are going through some kind of labor union negotiations. The tone of the meeting is "us vs. them". They (the customer) are going to want us (the team) to do things that we don't think is right or technically feasible. While, the customer is thinking -- "If we (the customer) don't get everything we want now, they (the team) will punish us in the end for missing requirements". Also, those customers were never seen again until the end of the project, at which time we went through further negotiations. Wouldn't it be much better to have a customer available (either directly or through a customer proxy) any time that you needed and to come to decisions on requirements throughout the project? There will always been some negotiation with customers but this should happen thorugh frequent collaboration instead of a few "negotiation" meetings.

Responding to change over following a plan - As a formal project manager, I can remember the motto - "Must keep this project on time and on budget". I also remember the evil words "Scope Creep" that indicated a project that "deviated" by introducing more scope in the middle of the project. People that introduced scope creep were considered evil (ok, not evil, but VERY bad) and were shunned for their actions. However, how important is following a plan if you aren't going to allow the customer to change their mind and provide feedback along the way. Part of "working software" is making sure the software is meeting the customer's needs. If the project isn't going to do that, it is really worth keeping it on time and on budget and avoiding scope changes at every turn? It is important to have some kind of plan as long as keeping to the plan doesn't forget to consider whether the customer's goals of the project are being met.

In a later post, I'll explore the principles that the Agile Alliance came up with that are the ideals behind agile development.

What kind of software would you like developed?

Tue, 04 Oct 2005 22:28:33 GMT

I'm curious and what to know. Say you have access to a software development shop and that told you that they could build any software product that you wanted. However, there are some options for you to consider. After reviewing your basic specifications, they sit down with you and come up with these alternatives:

Option 1: Build a product in 12 months that is simple and easy to use but only meets basic requirements. Upon completiion, this company will only fix bugs and provide minimal updates every six months; training through FAQs and some simple documentation; and support through basic email, forums, but no phone contact. The cost to you to pay for the development and ongoing support of this product is low.

Option 2: Build a product in 24 months that meets all the requirements including features that you might want to use in the future but is complex to configure and use. Upon completion, this company will fix bugs and provide enhancements or new functionality through a elaborate support agreement that includes frequent updates every three months; extensive training through onsite, manual and online and support through 24/7 phone,email and web. However, the cost to you to pay for the development and ongoing support of this product is high.

Option 3: Build a product in 18 months that is simple, basic and easy but an open architecture is developed that will allow others such as end users or other developers to make it as complex as they would like it through the development of addons and extensions. Upon completion, this company will fix bugs or enhance existing functionality only and provide moderate updates, training and support through a combination on in-house and community resources. The cost to you to pay for the development and ongoing support of this product is a little more expensive than Option 1. In addition, you will be on your own regarding support and future development of any additional functionality that is provided by third parties.

As an end user, which product would you pick and why? If you worked in this software organization, which product would you want to build and support and why?

UPDATE:

Here are my responses to each scenario, your answers may be different:

Scenario 1: Best for organization because it is easiest to staff and manage. Worst for end customers because as their needs change the software will become outdated.

Scenario 2: I believe that most development shops are set up this way currently. Best for end customers because all of their needs are met. Difficult to manage and staff for the long term given the amount of "product" to keep track of.

Scenario 3: I believe that most development shops will go to this model in the future. This is different than current open source in that there may be core product that is still managed by the organization but the hooks to the architecture to allow people to develop addons and extensions are made available open source. Best of both worlds for end users, development organization and those in the community interested in expanding the "core" product.

See Wayne's comment and trackback to his blog. He is an example of how other people in IT as seeing the value of moving in this direction. Take a look at such products as Google Maps, Mozilla Firefox, and Winamp have seen success with this approach.

Intro to Agile

Fri, 30 Sep 2005 16:55:01 GMT

Agile Software Development. If you work within the software industry, you are probably familiar with it and many of you may be developing this way. If you aren't, you may see this as a lean approach that could be translated into other industries. Before I get into what this type of software development looks like, let's discuss what has been traditionally been the way to develop software - waterfall.

Waterfall software development was named as such because work is performed in phases, with each phase needing to be completed before the next phase started. People will call these phases different things, but essentially they are the following:

Research -> Plan -> Design -> Implement -> Test -> Release -> Review

Each of these phases would concentrate on the entire scope of the project, and people could not move to the next phase until the previous phase was completed. Because of this, an emphasis on formal documentation was needed due to the amount of scope going through each phase and the fact that team members may not have participated in the previous phase and need information to start their part.

End users and other primary stakeholder are usually involved in the first couple of phases, then would not see the end result until it was ready for release. The review is usually some kind of project post-mortem with lessons learned. Depending on the overall timeframe of the project, people that participate in this review may have been on the project months ago, and many people cannot remember what the start of the project was like.

Several years ago, people started to get together and determined that there must be a better way. They also started their own "lightweight" or "agile" process that suited their environment. One of the better known methodologies out there is XP or Extreme Programming.

When I started seriously thinking about a different way for my organization to develop, my research quickly sent me to the Agile Alliance. What is the Agile Alliance? Here is what the site says:

In the late 1990's several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.
Early 2001 saw a workshop in Snowbird, Utah, USA, where various originators and practitioners of these methodologies met to figure out just what it was they had in common. They picked the word "agile" for an umbrella term and crafted the Manifesto for Agile Software Development, whose most important part was a statement of shared development values.
The Manifesto struck a chord, and it led to many new Agile projects being started. As with any human endeavor, some succeeded and some failed. But what was striking about the successes was how much both the business people and the technical people loved their project. This was the way they wanted software development done. Successful projects spawned enthusiasts.
The Agile Alliance exists to help more Agile projects succeed and to help the enthusiasts start more Agile projects. This particular page is to help people learn more about Agile software development. In keeping with the Agile emphasis on face-to-face communication, we urge you to visit a users group and talk to your peers about their experience.

Next week, I'll talk more about the Manifesto that was created by this group and priniciples around it and how they differ from traditional waterfall. Take a look around the site as you will find plenty of articles and links to various websites.

My Programmers Are ... WHAT?!?!

Wed, 24 Aug 2005 22:49:52 GMT

The post title from the Google Blogoscoped site sort of caught my attention -- "Why Good Programmers are Lazy and Dumb?" Lazy and Dumb?? Did I read that right? I knew I had to read more of this! Here are some exerpts:

"I realized that, paradoxically enough, good programmers need to be both lazy and dumb.

Lazy, because only lazy programmers will want to write the kind of tools that might replace them in the end. Lazy, because only a lazy programmer will avoid writing monotonous, repetitive code – thus avoiding redundancy, the enemy of software maintenance and flexible refactoring. Mostly, the tools and processes that come out of this endeavor fired by laziness will speed up the production.

Second (and I will elaborate a bit more on this because I find the concept to be less known than the first) a good programmer must be dumb. Why? Because if he’s smart, and he knows he is smart, he will:

a) stop learning
b) stop being critical towards his own work"

Please read more of the post as they give some examples of both. I guess I understand where they are coming from. I agree that programmers need to question everything and do the simpliest thing that can accomplish their goals. By doing that, you will know what is important, take complexity out of the equation...thus having a better product that is more maintainable while getting quicker results.

But, just you try calling your programmers lazy and dumb! Perhaps there is a better analogy to use here. Though, I have to admit that Google seems to get it done with their products!

Technology Never Sleeps

Mon, 18 Jul 2005 19:06:34 GMT

For those that don't understand technology and software engineering, there is an expectation that once a project has been completed that it is truly done. They also expect that there is a quick transition from development into production and are ready to move on to the next big thing.

For those that do understand, they realize that things are just beginning especially if you are introducing new technology, product or functionality that the company and customers have not used before. Why is there such a disconnect and how do you narrow the gap to ensure that enough attention is given to ensure the long term stability of both the technology and solution?

Here are some of the issues that are faced:

Customer changes - Customers can be fickle and have the right to change their minds. The challenge is to determine why they are changing their minds. Did their business change? Did we miss the mark on the functionality? Did we resolve the true problems or just some of the symptoms? Even with the best upfront requirements and design processes, there will still be changes in requirements as long as their are people who want to change their minds or their business.

Industry changes - You need to constantly be monitoring the industries that you serve. In this world of mergers, acquistions and consolidation, things can change quickly in an industry. If you were serving only a segment of a market, you can see that segment disappear quickly. Watch how you competitors are addressing the changes. Watch how other vendors are adapting. And make sure you are following suit.

Operational adjustments - Though you try to accommodate business operations of your customers as you develop solutions, sometimes the technology itself may require some changes in how your customers run their business and train their staff. Take, for instance, the first time that your customer might have used a mouse for a GUI application instead of just using a keyboard. That took some adjustments in how they interacted with the system (as well as knowing how to trouble mouse operations). These adjustments can also take place within your organization. Your sales people need to understand these new technologies and how to sell them to customers. Your technical support people need to understand how they work and when they can break so they can more effectively troubleshoot (and help prevent) customer problems.

Training adjustments - After customers start to apply and use the new technology and solutions, you will discover that they may be using it differently than expected, or that the information that you have provided isn't detailed enough.

Defect corrections - No matter how much testing that is applied, either inhouse or in the field through beta testing, problems will be found by customers. Hopefully, these problems are minor or cosmetic in nature, in which case you have more time to address. Otherwise, if they are more critical, you need the time to address the fixes.

Operating System changes - Unless you are using an older OS, you are well aware of how often the operating system changes - either with major new revisions or through minor fixes. Many of these minor changes have to do with hardware changes, security patches, or product stablity -- all are usually important to fix at some point. With major OS changes, there may be things that aren't backwards compatible, so you may have to change your products to support both new and older versions. If the changes are significant enough, you may decide to not support older versions of the OS at some point.

Hardware changes - Hardware changes at a tremendous rate, not just the main PC (server or workstation), but other peripherals - printers, pointing devices, keyboards, USB devices. Just like operating systems, there are major changes as well as patch releases to consider. Also, you need to consider how many kinds of printers you are going to support and "certify" through testing. And, you may need to consider when products are no longer supported either because they are obsolete or the technology is too old to be compatible with later technology.

Software Tool changes - This is an area that gets the least amount of exposure, mainly because it is not easily seen within the product or by customers. All of the infrastructure to build and support the product goes through changes - programming languages, development tools, database vendor like with hardware and operating systems. As these pieces change, the same decsions need to be made as you did with hardware and operating systems including the need to move technical tools to other platforms.

So what can you do to account for these changes in your product development?

Add tasks in your product development projects for changes - Where the changes can still apply to the general scope of those projects, you should account for changes in those projects.
Create product maintenance projects - Where there are a lot of smaller changes that relate to your products that can be grouped together in a project (and don't related to the scope of projects in the previous point), you should create separate maintenance projects to address the changes.
Create infrastructure projects - Where the changes aren't directly related to the development or maintenance of your products (such as operating system changes), you should create separate projects to address and incorporate the changes.
Educate customers and other non-technical people - Talk to your customers and others about the costs of incorporating new technology or solutions, both short-term (projects) as well as long-term (maintenance). The better they understand both the value of these changes (long term stability, better security, improve ease of use, etc.) as well as the costs mentioned above, the easier it will be to support the new technology and solutions in the future.

Who is our customer again?

Wed, 06 Jul 2005 17:54:24 GMT

When I first started this career, I knew who my customer was. That's because it was a department on another floor for the same company. I knew where they were located. I understood their business. And, I knew where to go if I had questions. Oh, how I miss those days. Even if that department was a little fickle on what they wanted, I knew eventually we would get it right and be able to move on to their next request. There was a great sense of accomplishment.

Later on, I was a consultant. For the same reasons, it made my job much easier as a developer knowing exactly who my customer was. If I needed to talk with them, we could call a meeting and have them sitting right across the table from us and work through issues, questions, etc.

If you are in either of these situations, consider yourself fortunate.

Now, imagine that you are building a car. One car. Not several models. Not several makes. One car. You can't afford to make more than one car. You may be able to eventually rebuild parts of the car, but otherwise you have to live with the car.

Now, imagine that you have several thousand people that you are trying to get to buy your car. They all have different needs. They all want the car to do something a little different. Sure, they all have the basic needs - it must have wheels, be able to move, steer and be safe but otherwise they want the car to do other things a little different. Some want power windows, some don't. Some want air conditioning, some prefer to roll down their windows to get fresh air. Some focus on the quality of the radio, or the trunk space, or which side of the car has the gas tank.

This is my world. We have a software package that meets the need of a particular industry. This industry consist of several thousand locations that along with several other competitors serves. Though they do have common needs about how they business is to operate and how the software package is to help them in that endeavor, they also run their business much differently in certain areas. They do that because it gives them competitive advantages. They do that because they may have different customers in which they serve. They do that because they may prefer a certain way of running business that they have become accustomed to. So I periodically have to ask....

"Who is our customer again?"

I do this because it is easy to forget about them, their needs, how they differ, how they are the same. They aren't somebody who is on another floor, or you can sit across from them in a meeting. There are too many of them, many that I have never met, many that I may never meet. Yet, I need to meet their needs. Because if I don't, there may be a chance our competitors will. As I mentioned in my previous post, we also have to be careful that this "car", our software package, doesn't become too complex in the process in order to reflect their needs.

This requires compromise, but also requires us to understand who the consensus is with our customers and what they believe is the right solution. This is easier said than done, but here are some things that we have done:

Customer segments - We have created categories of our customers based on size, location, whether or not they have a single store or multiples, and what kind of services they provide in their business. This gives us a simpler model to use.
Customer roles - As we look at functionality, we look at the roles that are played in a business. Owners, managers and staff are three kinds of roles. How are these roles different? How are they the same? How will each interact with our software?
Customer user group - As we develop a particular functionality, we try to determine a certain number of customers who we want to access before the software goes to everyone. We may use these customers in interviews early in the development process to gather requirements. We will ultimate use these customers as part of a "beta" to test and approve changes to our software before the rest get it.
Customer representatives - Because it can be costly and take the customers away from their business, we have people within our company that represent and "role play" as the customer. It may not be as good as having real customers, but it is better than nothing. Plus, many of these people either had a previous life in the same business or have direct communication with our customers.

The Complexity Pendulum

Tue, 05 Jul 2005 21:36:49 GMT

Can simplicity in software be achievable?

What do you have to give up in order to have that simplicity?

What are the costs of flexibility weighed against the cost of simplicity?

How does all of this relate to providing customer service to end users?

I struggle with these questions on a regular basis. Our company has established itself as having a reputation for taking care of the customer at all costs. If a customer has a unique need, we have provided an option to allow them to do it. If they want the functionality to behave a little differently than designed, we code in a special switch that they can turn on to have the application behave as they want. We talk in the language of switches, options, flags, etc.

Why don't we just change the functionality for all customers you might say? Good question. We make these assumptions:

That we must have provided the right solution in the first place and that this particular need is different.
That we cannot change this functionality because other customers will be impacted by this change.
That those other customers must be using and feel satisified with the existing functionality since we didn't hear from them.
That the functionality that we initially provided shouldn't change over time.

Have we validated these assumptions? No, because we don't want our customers to think that we didn't get it right the first time and that we don't want to validate that we might not have figured it out right the first time. All in the name of providing great customer service to the end users because we accommodate their needs without impacting them in any way that could be perceived as negative.

So, what does flexibility and customer service have to do with simplicity or complexity of the solution? The more straightforward you can provide a solution, with less "moving parts", the easier it is too maintain, validate(test), document and train the end user. The long term benefits of this simplicity means less ongoing operational costs. Less operational costs means that you can provide additional enhancements or solutions. Additional enhancements or solutions, with the simplicity of each solution, will create RAVING FANS!! (as Tom Peters! would say)

On the other hand, the more you do in the name of customer service to provide flexibility through options, switches, etc creates more operational costs. More operational costs means less enhancements and additional solutions. It also means that it will make it more difficult for end users to understand how to use all of those "moving parts" sometimes even to do the more simple things in their minds. Eventually, this will cause your end users to gain resentment for you in saying "What have you done for me lately?" and "Give me something that makes sense and that is usable".

I am sure that our company is not only in this struggle. Look at any software package and you can see that there is a very fine line between flexbility and simplicity. Some companies choose to focus entirely on simplicity, saying no often in order to focus on core functionality. Other companies choose to design their products in a way that the end user can define how simple or complex they want it: Look at Mozilla Firefox for instance in how they built the ability for plug-ins. Others have allowed their solutions to get complex over time by providing a lot of configuration options. It is the latter that I am concerned with.

What can be done? Here are some things that I am working on in my organization:

Validate assumptions about your end users. You saw our assumptions above. How do we know those assumptions are correct? When did we last validate them? Do we really understand our customers?
Make honest evaluations about current functionality. Did we really get it right in the first place? Has something changed in the marketplace or customer needs that we haven't accounted for? Determine what is really core to your solution verses nice to have accessories.
Avoid quick fixes and make time for the long haul. Put in the short term costs in order to have the long term benefits. Don't be tempted to go for the band-aid solutions and yet still have internal bleeding.

What is your usability test?

Mon, 02 May 2005 16:17:06 GMT

I had the opportunity to meet a few other business bloggers from the Portland Area on Friday, including a chance to meet the Buzz Bruggeman, CEO of ActiveWords and hear his story about the progression of his business and product. Bren over at Slacker Manager has a picture and some more information on our visit.

During our discussion, Dwayne from Genuine Curiousity said that the ActiveWords product met his "Sandwich Test". "Sandwich test, what's a sandwich test?", we all replied at the same time. Dwayne went on to say that his test for usability of a product is one that doesn't require a lot of switching back and forth between mouse and keyboard. In other words, he should be able to eat a sandwich in one hand and get around the application in the other.

Those words stuck with me, and made me wonder what other ways people think of usability with software applications. Number of keystrokes? Ease of using a pointing device? Use of wizards to educate you? Use of colors to show different kinds of information?

So what is your usability test when looking at a new application?

It's just that simple and it just works!

Mon, 04 Apr 2005 18:49:29 GMT

The agile development community has a best practice that I really like. It states that you should "Do the simplest things that can possibly work". In many ways, this saying has been around some time at least in the software development circles as the KISS principle (Keep it simple stupid!) but with a particular distinction -- IT MUST WORK!

Here's my definition of how you balance simplicity while providing a workable solution:

Don't sacrifice quality! Simple doesn't mean to do just things faster while skipping important quality checks along the way. To make sure that it works, it must be free of obvious defects.

Don't forget design! Simple doesn't mean hacking a solution, but what is simple to the end user. Design for simple functionality, which could require a complex implementation. You want simple design, look at the I-Pod Shuffle as a great example of simplicity to meet simple functionality.

Determine the least common demoninator of your user base. Every product has its first time novice users all the way to power users. Determine what every end user has in common and build for that. Obviously you won't develop every bell and whistle that your power users demand, but at least you know that what you deliver them and others works!

Staying competitive isn't just about having something more than your competition. Instead, start with having something better than your competition. With this simple approach, the focus must be on making sure that what you build impacts the bottom line better than your competitors - faster, more efficient, easier than other methods. In the end, isn't that what is most important to any customer?

Though this works well with development of IT solutions, this best practice can also apply to other aspects of your work such as process improvement. You would be amazed at how much overhead (and probably unused functionality, see my posting on this) is in your products and processes. Cut out the fat, and watch good things happen!

Estimating Software Projects: Is it worth it?

Mon, 14 Mar 2005 16:23:30 GMT

I have been in the software engineering industry now for almost 20 years. During that time, I have been surprised how much weight and expectation that management has on estimating projects. Typically, estimates are usually done early in the lifecycle, in order to come up with milestone dates.

What are these estimates based on? Usually, estimates are based on histrorical information - what you have done before. But how many software projects have you been on that was even close to what you have done before. Sure, the processes you do are probably simiilar, but the scope of those projects are always different. This is usually because the technology or the functionality that you are providing has not been done before, at least internally.

What information do you have at the time the estimates are done? Studies have shown that early estimates can have a wide range of precision because there is so much to learn. Yet, very rarely have I seen this communicated effectively with management. Once they see a date, they don't let go of it and just expect it to be hit. When that happens, the team is expected to hit the date at all costs even if it is unrealistic (which it usually is, especially if the technology or scope is unfamiliar to the team, or when there are many risks present). It is important to communicate to management of how much you don't know yet and when you plan on refining the estimates once you do know.

Are estimates worth it? Yes, as long as you understand that they are only used as a measuring stick to give an idea when the project will be completed. To be realistic on a project, you should gather new estimates on a regular basis and adjust milestones as appropriate...always educating management on why things have changed. This is one reason why I believe the agile movement is gaining momentum. By doing things in small iterations...not only can you provide better estimates but you are purposely redefining the milestones as you go. It works because it fits better the nature of software projects - that no project is the same as before unlike other industries.

Effectiveness of Software Features

Wed, 02 Mar 2005 01:23:52 GMT

Somebody handed these stats to me the other day that I thought were facinating. The information came from the Standish Group for XP2002. You find find a slide from the original presentation here.

In a large study of projects, it was discovered that:

45% of requested features were never used
19% of requested features were rarely used
16% of requested features were sometimes used
13% of requested features were often used
7% of requested features were always used

Amazing! What does this tell us? We aren't connecting with the end users that will use the software either through up front requirements or educating them on the new feature of the software. Either way, it's a real waste of effort. This should be a wake up call that we need to not second guess how we think end users will use the software, but go directly to the source for answers.