println it

Software blog about tools, builds and making it all work

Configuration Management position – can be done by anyone. Or can it?

Configuration Management was always my passion.

Somehow, I have always had this thing for builds. I don’t know why, it just happens to be that way. Over the last 10 years I have accepted various development positions, but very few were actually related to CM. Occasionally, I accepted CM tasks for the joy of it. But there were other reasons: most of the time any given software project is a mess at the CM level. Secondly, most projects on which I was working had no one assigned to this job.

Somehow, builds were supposed to take care of themselves. However in practice this does not happen magically by itself: builds are usually written by some poor guy, temporarily assigned to this task with one goal in mind; to get it over with as soon as possible. And I don’t blame him.

It made me think. What’s wrong here? Why does it happen this way, that very few projects can demonstrate well written build logic, where code is reused and there’s no place to copy-paste? Why do some managers believe there’s absolutely no need to assign a dedicated resource to deal with CM issues and it can be easily taken care of by any developer? And why when these managers experience a messy environment they are still not convinced otherwise?

Lots of questions. I’m going to try and answer some of them.

Why does it happen this way that very few projects can demonstrate well written build logic, where code is reused and there’s no place to copy-paste?

  • Most developers don’t like dealing with builds.

    This is the root of the problem. From what I have seen throughout my career, developers really don’t like dealing with builds. My guess is that it’s just plain boring and unexciting for most “normal” developers and you need to be somewhat abnormal to even remotely like it. What’s exciting about reorganizing files and watching closely how archives are actually created? Not much I suppose.

  • Build code is not shipped and you get no credit for improving it.

    Build logic is usually hidden from people’s eyes: “We just run this batch, we don’t really know how it works”. Since in most cases it is not the company’s business to sell their builds, the incentive to understand and improve them is very low. No customer gets to see the build in any event.

  • Excellent build tools have not existed for years.

    First there was Ant and there is Maven. Why were they not-so-good? They made two important things hard: custom or conditional build logic and reuse. Lack of built-in conditions made builds full of variables, flags and other awkward ways to simulate a simple "if". Lack of built-in reuse functionality made people copy-paste tons of code without ever trying to organize it.

    It was Ant which made an awfully wrong assumption: “There’s no place for "if"s and loops in your builds. Builds should be straightforward and go from A to B to generate a C, no questions asked.” Ant-Contrib was a savior here but reusing someone else’s code was still not an option; <macrodef>s only brought their own issues. Eventually, people turned to “copy-paste”, the best re-usability solution around.

    In Maven, conditional logic and reuse are also far from perfect. To a very limited extent conditional logic can be done with profiles and Ant/Groovy snippets while reuse is provided by <parent>-ing other POMs. I believe this is not flexible and elegant enough. In addition, Maven declarative nature makes any custom build tweaks extremely hard to implement: if there’s no attribute for what you want – you’re out of luck. The only sensible way to deal with all those limitations is to develop Maven plugins for every customized need you might have. Only then one can keep his POMs sane, not blowing to zillions of Ant tasks. But plugin development is not accepted readily by most developers so they usually turn to "maven-antrun-plugin" and .. here we have all Ant issues again.

    The problem with major build tools we have had for years is that they are mostly about straightforward build steps rather than about intelligent build logic. They lack basic build language which is nicely taken care of today by Gradle which has realized that builds are programs and deserve their own build language and build model. When Hans Dockter, the creator of Gradle explained that builds always have their own logic and should be taken as seriously as a production code, I was extremely happy to hear that. Finally, there’s a tool creator who gets it.

Why do some managers believe there’s absolutely no need to assign a dedicated person to deal with CM issues?

  • A dedicated CM resource is a wasted salary that contributes little to the company’s mission and profitability.

    This is so untrue but, unfortunately, that’s exactly how a CM position appears to some managers: a waste. Managers are unable to get answers to corporate questions such as “Can you demo what you do? Can you sell it?” In the absence of any commercially viable answers they wonder why a CM is required at all. Ironically, the most important aspect of any project gets the least attention. Projects with poor build logic are extremely hard to develop, maintain, test and release. It surely comes back to money and lack of understanding but some people fail to see it this way.

    If management believes that dedicated CM positions are not required then why do they keep an IT team? They can’t demo their work either. CM is required to keep things in order and to keep projects from falling apart.

    One possible way to provide a good explanation for the real value of a CM position is to show the cost of not having one. Usually it shows itself as delayed releases, people spending valuable time on activities that are not part of their jobs, notable time it takes to upgrade or change large portions of code, lack of a clear internal and external release processes and, as a result, a great deal of resources, energy and time wasted. And time is money.

    I have no doubt if I had to measure benefits of having a good CM it will always outweigh the mere salary.

Why can’t you split CM tasks evenly between all developers? It can be done by anybody, right?

This is a bad myth.

  • If you split responsibility nobody owns the problem and no one does anything about it.

    Software developers are far too busy to think of ways to improve builds. They can patch it somehow to remove the blocker out of the way but don’t expect them to give serious thought or make an effort regarding any build issue. Multiplied by developers’ aversion to builds in general, no wonder this approach produces little if any results.

  • CM position requires a very specific set of skills.

    As any other position, this one requires a person with a set of skills matching the job requirements. Why wouldn’t you split graphic design, backend logic or DBA tasks evenly between all developers? How would your application look if everybody was responsible for drawing icons, crafting RDBMS schema, and defining product scope? We surely know those tasks should be performed by specifically skilled people in an organization but, somehow, when it comes to CM “anybody can do it”. Why, oh why?

What are CM responsibilities?

It depends on your definition of CM. In some companies CM personnel are busy with baby-sitting ClearCase; they don’t get to write or think about actual builds. My definition of a CM’s responsibilities is as follows:

  • Choose, install and maintain build servers and tools.
    Consider upgrade or migration when a better build tool is being provided by the market.
  • Be deeply involved in and throughout the development process.
    “Outsider” view works much worse as it lacks knowledge required for making critical decisions, improvements and optimizations.
  • Monitor and “align” third-party library versions.
  • Implement build logic, educate developers and provide support where needed.
  • Constantly review company’s builds and search for overlaps, contradictions and unnecessary complexities.
    Develop solutions to prevent these.
  • Define and enforce version control system branching policies and guidance.
  • Define and enforce general build practices, product versioning, and release processes.

As one can see, the list is not short and it is far from being a naive one-liner like “Write Ant scripts” as some managers may conceive CM duties.

What are the skills required to perform the above duties well?

  • Genuine and natural ability to get things in order, always do the right thing and keep it that way!

    From my experience, a good CM person is a gatekeeper to prevent disorganization, lack of knowledge, interest and whatever it is that is preventing companies from keeping their projects organized, timely built and tested. The reality can be sad at times but a good CM professional fights it and doesn’t allow himself to give up and become careless. Left to their own devises projects may eventually derail into a clutter and a good CM is supposed to alert and block any such attempts.

  • Good development skills.

    I think it is crucial for CMs to be good developers because sometimes existing tools are inadequate and it requires a CM to develop new more appropriate tools. These tools can be Maven plugins, custom Ant tasks, Hudson plugins or some internal Web applications; whatever makes things tick. So good knowledge of Ant, Maven, Java and Groovy is very advantageous. Armed with good Java knowledge, CMs are able to understand and probably fix a lot of build failures, without resorting to external help. It’s about saving time and developing new solutions, whenever required.

  • Balls, curiosity, and good appetite for upgrades.

    Historically, companies are very reluctant to upgrade their build infrastructures, libraries and surrounding tools, like JDK. It can bring instability, failures and nobody really wants to deal with these. Indeed, it takes some will-power to undertake an upgrade in an organization. The upgrader is invariably responsible if anything fails and he is the one who needs to fix it. Consequently, one gets to witness organizations running with outdated versions of Ant, Tomcat, Spring and JDK for no other reason than the fear of upgrades. Good CMs should always be curious about new versions and insist on upgrades as part of their job irrespective of the resistance.

  • Good troubleshooting skills.

    One of the least exciting aspects of CM’s job is the obligation to deal with build failures. Sometimes it’s all too overwhelming – non-stop failures for various reasons: bad commits, lack of disk space, wrong OS, missing setups, and non-reproducible Hudson errors. Occasionally, it requires some emotional stability not to fall apart under the onslaught of problems but what it always requires is good troubleshooting skills: an ability to quickly understand what went wrong from looking at log files and analyzing all changes that were brought in recently. There are ways to deal with all of these, like bringing a system back into a working state and thereafter reverse engineering it to the point of failure, step by step.

  • Good communication skills.

    For any CM to successfully manage this otherwise challenging and sometimes thankless job requires excellent communications and interpersonal skills. It requires CM to convince some employees to change inadequate old habits and educating developers and users about the importance of good CM practices.

In conclusion, I would say that there is no substitute for a good CM in any serious organization that prides itself and has an above average development environment. I tend to compare the CM’s position to that of a building architect who is in charge of the overall construction design, its services, function, maintenance, security, and infrastructures. Someone needs to have a bird’s-eye view of all development activities. In theory Software Architects should be fulfilling this job but in practice it is CM role.

, , , ,

10 Responses to “Configuration Management position – can be done by anyone. Or can it?”

  • I find what you are saying here extremely useful and hopefully will be listed on by Management personal – I am not sure how you get them to listen but this looks like a well constructed and explained article which should (among other things) help.

    In addition I believe the role of RELENG / CM are nowadays not only maintaining the build process but also have a main role in Design & System scope considerations, these are crucial points which are often overlooked, and as the project matures one finds himself re-writing code due to bad / poor decisions during the projects bootstrap process.

    • I’m not sure you’ll get them to listen if they don’t already. Instead, I would try to show how damaging it is when there’s not enough attention to CM, how much time is wasted on builds and how many people deal with what they really don’t have to. Talk money, talk business. Simply put, it hurts to business not to have a CM staff and one can usually show it with $ and, eventually, share price.

      Yes, I agree on what you say about CM being involved in a planning phase. Normally, CM has a broader and better view on what’s company doing and he can provide some critical remarks about future projects as well.

      Yeah, it’s a tough job, no doubt about it.

  • I think it depends on the scale of the codebase and size of the team. Having a dedicated CM guy for small or middle sized teams is overkill. I think the most important thing (as you mentioned) is that somebody is the owner of the CM-problem. But thinking in terms of generalists in agile development developer itself can be owner of CM. I would only scale up to a dedicated CM position if the codebase or the number of projects is getting too big to be handled in parallel by on developer.

    • I’m afraid the drawback of not having such an “overkill” is that eventually you wake up with a project having poor or no build logic and then, when CM is added, he’s redoing the whole build from scratch – I saw it a number of times happening exactly this way. Or even worse, it is too late to change anything, people are too busy, and the project goes on with its ” old build problems” for years.

      I believe it is very beneficial to have a person *responsible* for CM, builds, general order, continuous integration builds, nightly builds, unit tests and integration tests in each team. Otherwise, all those critical issues can be neglected while everyone is busy with tasks at hand. It may not be a full-time job but *responsibility* is a key word here. Someone who’s on guard.

      A full-time CM position is for the whole organization, with a number of projects. So it may be a nice hierarchy where “company’s CM” is dealing with company-wide issues and “team’s CM” is doing the same but on a smaller scope. From my experience it works very well this way if people are cooperative and supporting.

      What do you think?

  • Didi says:

    I have been a developer for 6 years now and have been a build and configuration manager for a year and a half.
    Your article describes perfectly what i want to do in my job but i too am afraid that people think everyone can do this kind of work.
    Any suggestions on how to convince people to let someone do this job? There are also not many jobs on the market with a job description like that…

    • Hi Didi,

      When it ain’t broken it’s usually hard to convince someone to fix. Does it work? I mean, when nobody is doing CM what’s happening then? Where does it hurt?

      I can suggest to show people a *price* they’re paying for not having a CM: it can be delayed releases, no releases at all, extra-long build time where engineers have nothing else to do but to stare at the screen, wasting corporate time .. In many cases the price is wasted time and lack of automated processes like CI/nightly/release builds and integration tests.

      If you were to explain me why do you need to let someone to do this job – what would you say? That’s probably the first question any manager will ask when approached with a proposal to fill in a CM position.

      I would also ask the following questions:
      - Do you have CI/nightly/release builds?
      - Do you have integration tests?
      - Do you have branches?
      - How much time does it take to release a version?
      - How much time does it take to release a version *from a branch* ?
      - What Spring/Tomcat/JDK/whatever version is used in all projects? How many changes in how many places do you need to make in order to upgrade or downgrade it?
      - When there’s a compilation error how long does it take to discover and fix it: minutes, hours, days, weeks? Same about integration tests

      Normally, you can show that to deal with all those issues is highly beneficial for a company, its success and customers satisfaction and that’s exactly what CM position is for.

      • Didi says:

        Hi Evgeny,

        Thanks for your reply.
        It’s very frustrating because for the most part I can answer yes to all of your questions but in our development team we adopt the Agile Methodology. This is the main reason why this job is not assigned to just one person. Here, they want every developer to be able to do everything related to software development, also to spread knowledge, which is of course not possible.
        But trying to convince them of that is not easy.

        • I see. I guess it can be frustrating indeed when something so obvious isn’t obvious to other people. Probably, they don’t see it as a problem that needs to be fixed. Probably the situation is Ok even without a dedicated CM resource, though I doubt so.

          In this case I would probably try just do the right thing and improve what needs to improved, without being officially assigned. Once people see how different the situation is they will appreciate the change and will understand better what CM is about. If they object to what you do – you can start asking questions like “Do you think it’s better if we have Xyz? (nightly builds, integration test)” to which they will probably agree.

          So if talking and convincing doesn’t really work then doing probably will.

    • Regarding jobs that fit this description, you can probably try “Configuration management” group in LinkedIn: or any other CM/ALM-specific resource. I’m sure there are plenty of them.

  • [...] or organization, has to be honest about investing in it for real. This situation is similar to Configuration Management: customers don’t buy tests so the incentive to create, improve and maintain them is [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>