Goldin++

Configuration Management was always my passion.

Somehow, I have always had this thing for builds. I don’t know why, it just happens to be that way. Over the last 10 years I have accepted various development positions, but very few were actually related to CM. Occasionally, I accepted CM tasks for the joy of it. But there were other reasons: most of the time any given software project is a mess at the CM level. Secondly, most projects on which I was working had no one assigned to this job.

Somehow, builds were supposed to take care of themselves. However in practice this does not happen magically by itself: builds are usually written by some poor guy, temporarily assigned to this task with one goal in mind; to get it over with as soon as possible. And I don’t blame him.

It made me think. What’s wrong here? Why does it happen this way, that very few projects can demonstrate well written build logic, where code is reused and there’s no place to copy-paste? Why do some managers believe there’s absolutely no need to assign a dedicated resource to deal with CM issues and it can be easily taken care of by any developer? And why when these managers experience a messy environment they are still not convinced otherwise?

Lots of questions. I’m going to try and answer some of them.

Why does it happen this way that very few projects can demonstrate well written build logic, where code is reused and there’s no place to copy-paste?

• Most developers don’t like dealing with builds.
  

  This is the root of the problem. From what I have seen throughout my career, developers really don’t like dealing with builds. My guess is that it’s just plain boring and unexciting for most “normal” developers and you need to be somewhat abnormal to even remotely like it. What’s exciting about reorganizing files and watching closely how archives are actually created? Not much I suppose.

• Build code is not shipped and you get no credit for improving it.
  

  Build logic is usually hidden from people’s eyes: “We just run this batch, we don’t really know how it works”. Since in most cases it is not the company’s business to sell their builds, the incentive to understand and improve them is very low. No customer gets to see the build in any event.

• Excellent build tools have not existed for years.
  

  First there was Ant and there is Maven. Why were they not-so-good? They made two important things hard: custom or conditional build logic and reuse. Lack of built-in conditions made builds full of variables, flags and other awkward ways to simulate a simple "if". Lack of built-in reuse functionality made people copy-paste tons of code without ever trying to organize it.

  It was Ant which made an awfully wrong assumption: “There’s no place for "if"s and loops in your builds. Builds should be straightforward and go from A to B to generate a C, no questions asked.” Ant-Contrib was a savior here but reusing someone else’s code was still not an option; <macrodef>s only brought their own issues. Eventually, people turned to “copy-paste”, the best re-usability solution around.

  In Maven, conditional logic and reuse are also far from perfect. To a very limited extent conditional logic can be done with profiles and Ant/Groovy snippets while reuse is provided by <parent>-ing other POMs. I believe this is not flexible and elegant enough. In addition, Maven declarative nature makes any custom build tweaks extremely hard to implement: if there’s no attribute for what you want – you’re out of luck. The only sensible way to deal with all those limitations is to develop Maven plugins for every customized need you might have. Only then one can keep his POMs sane, not blowing to zillions of Ant tasks. But plugin development is not accepted readily by most developers so they usually turn to "maven-antrun-plugin" and .. here we have all Ant issues again.

  The problem with major build tools we have had for years is that they are mostly about straightforward build steps rather than about intelligent build logic. They lack basic build language which is nicely taken care of today by Gradle which has realized that builds are programs and deserve their own build language and build model. When Hans Dockter, the creator of Gradle explained that builds always have their own logic and should be taken as seriously as a production code, I was extremely happy to hear that. Finally, there’s a tool creator who gets it.

Why do some managers believe there’s absolutely no need to assign a dedicated person to deal with CM issues?

• A dedicated CM resource is a wasted salary that contributes little to the company’s mission and profitability.
  

  This is so untrue but, unfortunately, that’s exactly how a CM position appears to some managers: a waste. Managers are unable to get answers to corporate questions such as “Can you demo what you do? Can you sell it?” In the absence of any commercially viable answers they wonder why a CM is required at all. Ironically, the most important aspect of any project gets the least attention. Projects with poor build logic are extremely hard to develop, maintain, test and release. It surely comes back to money and lack of understanding but some people fail to see it this way.
  

  If management believes that dedicated CM positions are not required then why do they keep an IT team? They can’t demo their work either. CM is required to keep things in order and to keep projects from falling apart.
  

  One possible way to provide a good explanation for the real value of a CM position is to show the cost of not having one. Usually it shows itself as delayed releases, people spending valuable time on activities that are not part of their jobs, notable time it takes to upgrade or change large portions of code, lack of a clear internal and external release processes and, as a result, a great deal of resources, energy and time wasted. And time is money.

  I have no doubt if I had to measure benefits of having a good CM it will always outweigh the mere salary.

Why can’t you split CM tasks evenly between all developers? It can be done by anybody, right?

This is a bad myth.

• If you split responsibility nobody owns the problem and no one does anything about it.
  

  Software developers are far too busy to think of ways to improve builds. They can patch it somehow to remove the blocker out of the way but don’t expect them to give serious thought or make an effort regarding any build issue. Multiplied by developers’ aversion to builds in general, no wonder this approach produces little if any results.

• CM position requires a very specific set of skills.
  

  As any other position, this one requires a person with a set of skills matching the job requirements. Why wouldn’t you split graphic design, backend logic or DBA tasks evenly between all developers? How would your application look if everybody was responsible for drawing icons, crafting RDBMS schema, and defining product scope? We surely know those tasks should be performed by specifically skilled people in an organization but, somehow, when it comes to CM “anybody can do it”. Why, oh why?

What are CM responsibilities?

It depends on your definition of CM. In some companies CM personnel are busy with baby-sitting ClearCase; they don’t get to write or think about actual builds. My definition of a CM’s responsibilities is as follows:

• Choose, install and maintain build servers and tools.
  Consider upgrade or migration when a better build tool is being provided by the market.
• Be deeply involved in and throughout the development process.
  “Outsider” view works much worse as it lacks knowledge required for making critical decisions, improvements and optimizations.
• Monitor and “align” third-party library versions.
• Implement build logic, educate developers and provide support where needed.
• Constantly review company’s builds and search for overlaps, contradictions and unnecessary complexities.
  Develop solutions to prevent these.
• Define and enforce version control system branching policies and guidance.
• Define and enforce general build practices, product versioning, and release processes.

As one can see, the list is not short and it is far from being a naive one-liner like “Write Ant scripts” as some managers may conceive CM duties.

What are the skills required to perform the above duties well?

• Genuine and natural ability to get things in order, always do the right thing and keep it that way!
  

  From my experience, a good CM person is a gatekeeper to prevent disorganization, lack of knowledge, interest and whatever it is that is preventing companies from keeping their projects organized, timely built and tested. The reality can be sad at times but a good CM professional fights it and doesn’t allow himself to give up and become careless. Left to their own devises projects may eventually derail into a clutter and a good CM is supposed to alert and block any such attempts.

• Good development skills.
  

  I think it is crucial for CMs to be good developers because sometimes existing tools are inadequate and it requires a CM to develop new more appropriate tools. These tools can be Maven plugins, custom Ant tasks, Hudson plugins or some internal Web applications; whatever makes things tick. So good knowledge of Ant, Maven, Java and Groovy is very advantageous. Armed with good Java knowledge, CMs are able to understand and probably fix a lot of build failures, without resorting to external help. It’s about saving time and developing new solutions, whenever required.

• Balls, curiosity, and good appetite for upgrades.
  

  Historically, companies are very reluctant to upgrade their build infrastructures, libraries and surrounding tools, like JDK. It can bring instability, failures and nobody really wants to deal with these. Indeed, it takes some will-power to undertake an upgrade in an organization. The upgrader is invariably responsible if anything fails and he is the one who needs to fix it. Consequently, one gets to witness organizations running with outdated versions of Ant, Tomcat, Spring and JDK for no other reason than the fear of upgrades. Good CMs should always be curious about new versions and insist on upgrades as part of their job irrespective of the resistance.

• Good troubleshooting skills.
  

  One of the least exciting aspects of CM’s job is the obligation to deal with build failures. Sometimes it’s all too overwhelming – non-stop failures for various reasons: bad commits, lack of disk space, wrong OS, missing setups, and non-reproducible Hudson errors. Occasionally, it requires some emotional stability not to fall apart under the onslaught of problems but what it always requires is good troubleshooting skills: an ability to quickly understand what went wrong from looking at log files and analyzing all changes that were brought in recently. There are ways to deal with all of these, like bringing a system back into a working state and thereafter reverse engineering it to the point of failure, step by step.

• Good communication skills.
  

  For any CM to successfully manage this otherwise challenging and sometimes thankless job requires excellent communications and interpersonal skills. It requires CM to convince some employees to change inadequate old habits and educating developers and users about the importance of good CM practices.

In conclusion, I would say that there is no substitute for a good CM in any serious organization that prides itself and has an above average development environment. I tend to compare the CM’s position to that of a building architect who is in charge of the overall construction design, its services, function, maintenance, security, and infrastructures. Someone needs to have a bird’s-eye view of all development activities. In theory Software Architects should be fulfilling this job but in practice it is CM role.