The Jolt Awards are the major industry award for software development tools (compilers, libraries, etc.). One problem we face every year is proper classification of tools. Traditionally, we try to refine / fine tune the previous year's categories (Development Environments; Libraries, Frameworks, and Components; etc.). Problems arise frequently balancing the number of products in a category (20 entries in one category, 3 in another), when clearly competitive products end up in different categories (happens all the time with categories "Web Development Tools" versus "Development Environments"), and when a product cuts across categories.
Brainstorming yesterday, we wondered if it would not be better to generate the categories dynamically. One idea was to use checkboxes for predefined activities ("defect tracking," "code generation," "GIS mapping") and use some form of entropy measure to divvy them up into our 12-or-so categories. Easy enough mathematically. Another, more dramatic idea, was to create a tagging system for software tools and see if we could come up with a more dynamic view. The main challenge we see is that it seems like a small world: there are only a few hundred tool releases every year and it's difficult to imagine many people becoming engaged in the task of tagging them.
Do the Web 2.0 dynamics of distributed collaboration apply to small numbers? A del.icio.us for software development tools?