Software reuse on a budget
17-Jul-2006Most organizations wake up one day and wonder how much more competitive they would be if they never wrote the same code twice. Many organizations undertake significant organizational changes to implement their new found software reuse religion. Generally, reuse initiatives like these are enthusiastically evangelised, resisted by staff, provide dubious economic benefit when they’re in operation, and take years of investment before they are renamed according to the fashion of the day and re-launched.
At the moment web services (WS) and service oriented architecture (SOA) define the vocabulary of software reuse. In previous years it was object models (distributed and plain-ol’), product line engineering (PLE) or component-oriented architecture leading the charge to re-usability nirvana. My software reuse strategy isn’t an interface definition language or a step-by-step methodology. It is a policy suggestion.
I think the easiest and most effective way to achieve effective software reuse for any organization is to institute this policy:
Every software project must publish all of its development artifacts — code, test, requirements, help, training, bug reports and design information — such that any developer in the organization can search and view any development artifact or identify and contact the people working on the project without restriction. For bonus marks, we will standardise on a configuration management, bug and task tracking, development and documentation environment that makes looking at any other project’s artifacts as close to zero-cost as possible.
This may not be a revelation for some organizations. Some developers already enjoy unfettered access to their colleagues’ project files and can search across the entire organizations’ software assets for code, design and knowledge they can make use of. The tendency for large organizations is to Balkanize and tool up in such a way as code and design becomes invisible outside a project. I’m advocating that no matter what the political structure — make sure development environments present no barrier to searching and copying.
I think too much is made of the value of generating reusable software that is packaged in nice, neat, black boxes lined with soft archetype-affirming design documentation, and set to rest in a software asset mausoleum. Standardised look and feel and avoiding embarrassing contradictions in different software produced by the same organization are just some good reasons to formalise some reuse rules and methodologies, but the way to start down that road is to make sure there is a culture of cooperation and sharing. If you are a CTO looking to implement change to foster software reuse and you can’t see and search all of the software you’re responsible for, I doubt your reuse initiative is going to be more than a PR exercise anyway.
While I think there is merit in approaches advocated in many texts,I think that the strategy that is most likely to succeed is the one that does not require more investment than an important and influential person writing down that from now on no project shall have code or any other artifact that is inaccessible to any other developer in the organization. All project artifacts should be available to be indexed and searched by the organizations’ chosen (preferably competent) search engine.
The benefit of this policy is that reuse will occur. Developers like to branch and merge more than they like to use half-thought-out libraries on a different schedule full of set-in-concrete code designed by people that never thought your specific problem would be the one they needed to solve. Fertile open source projects like Linux are surrounded by mini projects that branch from the “main base” and contribute back the benefits in a time-frame that suits the main and branched bases, and opens up opportunities for future branched projects. With an open internal development environment developers can find and talk to each other, search and review the available code and design, copy what they need, share experiences, and avoid unnecessary rework. All of this is possible without any explicit software reuse initiative.
The drawbacks? A rogue developer will be able to release all of your organizations’ code (and not just those projects she was working on) to the known universe when things go sour for you and her. So sue Sue.
According to the signature image at the NASA Goddard Space Flight Center’s software reuse site “It should be as easy to find a good quality reusable software asset as it is to find a book on the Internet.”
John Udell cites an InfoWorld Programming Survey from 2003, where the biggest obstacles to software reuse perceived among developers were:
- “Effort required to design software for reuse” — 29%,
- “Lack of awareness of which software is available to reuse” — 28%,
- “Effort required to learn and effectively apply software available for reuse” — 21%,
- “Programmer disinclination to package software for reuse” — 10%, and
- “Effort required to package software for reuse” — 7%
- “Other” — 5%
Clearly an open internal development environment is applicable to four of the five points here. Perhaps these respondents already have completely open work environments and were looking beyond sharing and cooperation for the perfect reuse technologies.
If not? Perhaps making software available for reuse that cost nothing to design for reuse will help. Perhaps making all software easily searchable and available for reuse at no cost will help. Perhaps never asking programmers to package software for reuse will help.
Perhaps.
NASA's dream has come true. In the Java world, an
Alan Green | 17-Jul-2006NASA’s dream has come true. In the Java world, an astonishing amount of inter-company reuse goes on through Apache Jakarta, and SourceForge. Add to that bits and pieces from Java.net, and project websites reachable from Google and it’s amazing how much code a Java programmer don’t have to write.
The story for Perl is even better. It’s almost simpler to find a CPAN module than it is to find a book on Amazon, because with Amazon, you have to start a browser first. Maybe NASA hasn’t heard of Perl.
All this goes to support the mechanism you describe: the code is findable, so developers find it. I find it interesting, though, that the majority of successful reuse is not reuse of domain models or business functions, but rather with libraries of generic functionality, frameworks and interfaces to the real world.
The missing part of this hastily written mind-dump is that
Chris | 18-Jul-2006The missing part of this hastily written mind-dump is that I seek to emulate the seamless reuse in open source communities within a closed-source organization. The fork, evolve and assimilate approach is just the way it is done, and it’s not called “reuse” it’s called open source. I suspect this practice is ignored in reuse studies because it’s not concerned with packaging or other nominal code reuse practices.
In line with your last point — I think the majority of reuse is not explicitly recognised. When you write Java or C or C++ you’re reusing a library codebase, and the code generated is reused. When you run on an operating system, you’re reusing a substantial codebase. When you develop within a development environment you’re reusing software. When you target web browsers you’re reusing code. These forms of reuse are taken for granted. I think the concern of most organizations is to ensure that their own beautiful code is reused.
I think that code reuse is generally thought of as a problem that needs solving. I think code reuse needs to be thought of as one many methods of increasing productivity, and one that has the potential to decrease productivity if applied injudiciously.
Uhoh, now you've done it! Let the rant commence: The problems
Richard | 19-Jul-2006Uhoh, now you’ve done it! Let the rant commence:
The problems I see with the fork-and-paste style of code reuse are that: a) telling all the users of a particular class that they’ve got a serious bug, and that they need to upgrade/merge, b) writing code that depends on brittle libraries, c) terrible support for class versioning in most languages.
a) is caused precisely by the no-holds-barred approach to reuse: I can and will copy whatever I can lay my handds on, only to find that you’ve let a security flaw through, and that flaw has infected my code too. Unless I carefully isolate the rest of my app from your library, I could be in for an even more serious flaw in my own application. If your code has been abandoned (as is often the case with Sourceforge, and even some Apache projects), then I won’t even know there was an issue. Resolving the ability to contact me when you discover the problem requires some kind of notification infrastructure, perhaps with an automatic subscription as soon as I download the code.
b) is caused by the number one imperative of a commercial software company: to make money from a product/solution/box/cd as soon as possible, for as long as possible. The developers in the company are strongly encouraged to make their designs as simple as possible, yet still flexible, but only for the domain they’re writing their application for. After all, they’re not expecting their technician’s desktop tool code to wind up in an end user’s core business application.
If the developers of the core business application are careful, they’ll unit test, and functional test the reused code just as if they’d written it themselves, but it’s more likely they’ll just expect it to work, just like we currently do for an Operation System, a programming language’s built in APIs, and so on (well mostly, anyway: the Java Bug Database makes it clear that a long history of use doesn’t make even the simplest APIs defect free).
If the core business app developers are even more careful, they’ll budget time to fix some number of defects in the reused code, ensuring that their changes filter back to the original developers (if they’re still around, or care). However, the more predicatable scenario is a cost and schedule overrun while the team scramble to shore up the foundations of their application, or even delivery of an almost correct (ie deeply flawed) release.
c) is caused by the simplest form of reuse you’re likely to see: I keep the same namespace/package for the reused code, but I patch in a change to the code. You also make a change to the reused code (hell, it’s probably ‘your’ code in the first place). Yet another person tries to reuse your code and mine, and they run into the source-level version of DLL hell. They’re now stuck with the task of migrating my code to the new version of yours, with the possibility that I can then change mine further without accepting their or your changes to your code into my sourcebase. This would mean that they’d have still further work to do to support the two systems, even though reuse is supposed to have avoided this whole problem. An excellent example of how badly this can go is how Java rolled the Xerces XML parser into the standard libraries, leaving users of more modern versions of Xerces with no easy migration strategy.
Languages like Ruby are an improvement in this area, as it’s RubyGems package management system (interesting. Packaging for reuse, eh? Fancy that…) can allow different versions of the same component to be loaded at the same time. Unless you’ve done some amazing magic with class loaders, you’re going to have a lot of trouble doing anything similar in Java. A Servlet container like Tomcat’s classloader heirarchy is a good example of the sort of magic that’s required, and even that has issues (eg, see http://www.qos.ch/logging/thinkAgain.jsp).
Frankly, I don’t see affordable code reuse happening across projects within big companies until developers are helped and encouraged to update their code for other developers within the same company. Software that’s lost all of its developers (thanks to the low, low average stay for a developer at any one job, and management’s fine habit of moving people away from maintenance of ‘finished’ products as far and as fast as they can) should not be relied upon for building reliable systems. Open source software that isn’t changing isn’t good for long term use (9 out of 10 dentists agree with me, so it must be true), and the same is true for commercial software.
You need coordinated development of libraries and applications built with them. You need mechanisms for running two versions of the same library side-by-side. You need real component software.
[BTW, despite the hype behind oh so many tools, I don't think the world has yet had a component oriented system (as if you hadn't guessed). We've had some component-like systems (SOA is in this category, as do EJBs, Servlet Warfiles, Active X and friends, OpenDoc), and systems that claimed to be component-capable (Hi, CORBA, I'm talking about you!), but they're as fun to use as C is when it comes to object programming: you *could* develop C libraries using objects, but it requires creating your own private world of calling conventions, naming guidelines, and a vast deal of paper to paper over the cracks between this and the standard C programming idioms. Don't even try and use your C objects with someone else's C objects.
Believe it or not, Microsoft's .NET is the closest to a component-oriented system that I've seen. However, there's much more that needs to improve before we'll be in a component-oriented world.]
I agree with you on A, B, and C.
Chris | 25-Jul-2006I agree with you on A, B, and C. Hell, this is such a good rant I should promote it to content :)
On A - multiplying bugs. I agree this is a problem, but given a (false) choice to do nothing for reuse, and allow branch and merge open slather I’d opt for the latter. Whether it’s reuse with no maintainer, or striking out on your own to reproduce the mistakes of the past, you’re in a similar situation. The best solution is a good fast central component with versionable interfaces and … [insert your own reuse nirvana here]
On B - Yes, and this list of problem will occur in a variety of reuse strategies.
On C - money quote:
I agree. Any project manager worth their salt utters the words “eliminate dependencies” meaning - eliminate dependence on things I can’t control. Actually, I’d expand that to things that aren’t predictable. Companies need to make reuse as deterministic as possible, and that means commitment to support internal users, and not just your widget manufacturing application that you originally wrote that library for. Without that assurance, open slather, searchability and the kindness of strangers is all ya got.