Two fascinating articles. I agree with Marcus Ranum, it is interesting how in Feynman’s “Personal observations on the reliability of the shuttle” highlights the good quality of the software. Given how software is usually so bad, how did this happen? Well, it seems that NASA recognised that testing is expensive and hard, and that ‘have you tried rebooting’ is not an adequate answer to a problem. Good testing takes time, money and aggressive pursuit of something better than ‘good enough’.

Those boys at NASA clearly understand software testing. After all, it ain’t rocket science.

Control is a closed loop process.

You have some input, something happens, you look at the output and adjust your input again. This is implicit in quality procedures everywhere. This is what testing is.

Over the last few days I’ve been asked to estimate how long various bits of work for a potential customer will take. I’ve tried to make good estimates, but the truth is that I have never seen a comparison of how long was estimated at the beginning of a project, and how long was actually taken.

In other words, I still only have my gut feeling, my perceptions, to guide me as to how accurate I think my estimates are, despite the fact that this is a clearly measurable metric. The control loop is still open.

At the moment I provide estimates like “I think it’ll take about 10 days, but I’m not really very sure”. Wouldn’t it be better to know “Andy normally under estimates by and average of 10%, with a standard deviation of 10%”. Then if, for example, I estimated 10 days, we’d know that it’d actually 10-12 days, but only 68% of the time, or 9-13 days 96% of the time.

Perhaps this exercise is performed at a management level already – but as the developers are being the ones asked to estimate for technical details, it’s us that need our accuracy fed back for those aspects.

Now, I realise that there’s always going to be a human aspect to providing an estimate, that projects often aren’t that clear and easy to cut up, and that often we’re working with new things that we simply aren’t sure about. There’s also a cost-benefit question as to the effort in feeding back accuracy data. But I think that there are broad trends we should be able pull out.

I guess what I’m thinking is that we should have a process for optimising our estimates. I guess that this would involve examining statistical methodology. The best we’ve got at the moment is an informal “well that look longer/less time than I thought” after we’ve completed some work – and over projects that can span years, you just lose track.

It does strike me that it should be straight-forward to collate this data, and then feed it back at the end of the project. Close the loop. Regain some control.

Some useful links to things to remember in testing in Coding Horror. I agree; developers are poor testers, and the guy who wrote it is the worst person to do testing.

I said I’d follow up my earlier wondering into bespoke software vs huge applications, after I’d thought a bit about why companies don’t seem to like small, bespoke apps. Continue reading →

Cost aside, given the choice between a suit ‘off the shelf’ and one ‘made to measure’ in Saville Row, most people would go for tailored suit every time. So why doesn’t that happen in software?

Okay, so I know I just made that comparison so that it wasn’t like-for-like by excluding cost, but it seems to me that tailors are a good analogy – and that the shopping habits of the well dressed man and the well equiped company are quite different.

Consider the suit. You can buy one ‘off the shelf’. It won’t fit as well as one made specially for you, but it might fit well enough. Sure, the legs might be a little long, and it’s a little broad across the back, but hey, it’s a lot cheaper.

Another option would be to get a bespoke suit. Bespoke it a wonderful word that makes me think of bicycles, but here I mean it as ‘made to measure’. You suit will be unique, comfortable and well fitting, and it’ll cost much, much more. I know I’ll have made it when I get a bespoke suit.

The third option is to get the cheap suit, and have some alterations done. That is, you get someone to make small changes to the cheaper suit, so you get a better fit, and it still costs less.

Software is the same. It can be made for you, you can buy it off the shelf, or you can buy it and hack it until it does what you want it to. (I mean ‘hack’ in the old school way, not the illegal way).

Most people, when they buy suits, just keep shopping until they find one that fits. The luck few (or those unlucky enough to have unusual measurements) get them made for them.

Companies and software doesn’t seem to be the same though. They’re in all shapes and sizes, and as profitability often depends on good software enabling your processes, they all want suits that fit very, very well.

Lucky companies can find a bit of software that fits just perfectly, and within specialised fields, there are products like that. This is great – the suit is cheap and fits well. However, most companies have oddities that mean, for certain applications, no software fits very well. I guess, companies have less regular measurements than people.

So, now they’ve got a choice – bespoke, or get something and customise it. Actually, here the analogy breaks down a bit – there is a third option. Some software is built to be highly flexible and adaptable, so that it can fit many jobs. I kind of think of this as a ‘shrink to fit’ suit – I know, it’s straining. Humour me.

Okay, so, imagine we get some software and decide to customise it. Customisation costs, and it means that what you’ve got isn’t exactly the same as anyone else. You need someone to figure out what needs done, and make the changes for you. The constraints of the system can make some of those changes hard. Indeed, if the software fits poorly enough, you can end up having to rip the whole thing apart and rebuild it to do what you wanted to (Been there, done that). Indeed, you might as well have had it made bespoke.

Alternatively, you could use a ‘shrink to fit’ option. I think Microsoft SharePoint 2007 is a great example of that. It’s very flexible, with all sorts of neat bells and whistles, so it will fit many, many different needs. The problem is configuring it, shrinking it so that it fits well is also made very hard by this. Again, you’re into the realms of needing a specialist to help you. And you might not even need all those bells and whistles. At the end of it all, it could be as bad as having someone dismantle it.

So finally, we come to bespoke. You get someone to make it specifically for you. It does the what you want it to, and only that. You’re not paying for bells and whistles, and as it is designed to do just what it does, it is simpler than the other options. Sure, the tailors costs are a little scary – but done right, you don’t get all sorts of extra cost creep in later.

I keep ‘seeing’ bespoke solutions. I’m not working on them – I’m probably hacking around with some third-party application again, trying to make it do what the customer wants – and this vision comes into mind of what I could build to solve just that problem. Normally, they solve just a single problem – but do that well. And I wish I could do them. It seems to me that development is become faster, and frameworks are becoming better. Look at Ruby on Rails – it gives so much for free, you can really build an application quickly in it. Even (though I hate to admit it) Visual Studio can give you a lot, sometimes – although it is collapsing under the weight of trying to be all things to all people.

So my question is this – why do companies seem so keen to stay away from an application that just does what they need it to, and yet happy to buy something off-the-shelf and then have to change it to do what they want it to?

I guess I’ll consider that later…

When you say ‘soup-to-nuts’, we don’t know what it means. Frankly, it sounds like a potentially painful accident (depending on the temperature of the soup).

I looked it up. It means ‘end-to-end’. Why not use ‘end-to-end’ then? We all need to avoid strange localisms in our language. I mean, I’ve never finished a meal with nuts. Cheese or coffee, yes, but never nuts.

And the word ‘doable’ is not a word – we’ve already got a word for that anyway, and it is possible.

I’m guilty too – I used “elephant in the room” for a bit of humour. Completely confused my Zimbabwean colleague with that phrase. Presumably, elephants in rooms really was more of a problem for him.

So, according to ZDNet, Security expert Howard Schmidt wants coders to be held responsible for vulnerabilities in their code. This is REALLY dumb.

He gets one thing right – I’ll give him that. Most developers don’t have an adequate idea of what security entails, and training in this is, at best, extremely rare. There should be more of that, both at university and in the job – attacks evolve, after all.

But making developers responsible? When they don’t have the authority to control the product? Management choose what features are ‘in’ or ‘out’, project times scales, budget, etc.. I’d love to produce better code, but my boss will reject my ‘It’ll take twice as long and be 3 times as expensive’ – and rightly so. Continue reading →

An interesting article about ‘Web 2′. A bit buzzwordy, but it’s point is valid – the web is moving away from a book based ‘page’ paradigm towards a ‘data aggregation’ one. Instead of visiting specific pages, grab the data you need and put it onto one.

It’s interesting, and exciting, and although user discomfort with change will slow it down, we’d best be ready for it.

“Programmers Need To Learn Statistics Or I Will Kill Them All” – an article about statistic and programme testing. You know, load testing and all that jazz that you know you’re supposed to do.

Recently I was tasked with doing a heap of load testing, and we had all sorts of problems – all related, really, to confounding. I didn’t (and don’t) feel I know enough to design a load test, and the tools we were using were, well, limited. It’s interesting to note that I came to the same conclusion – that the data throughput was the most important metric.

However, the reason all these things talk about the number of users isn’t because programmers want it – rather, management and customers want to know how many users will it support. Telling them that you got an average data throughput of whatever is a better measure, but it doesn’t really mean anything to them – whereas could support 25 concurrent users does.