Roadmap

Release Stream

0.1.0 (complete, 04 Oct 2007 )

First real internal release, developing process, attempting Fedora run

0.2.0 (complete, 12 Oct 2007)

Added spec and mimetype agents, scheduler performance boost; enable Fedora analysis

0.3.0 (complete, 17 Oct 2007)

Fixed serious DB problems

0.4.0 (complete, 26 Oct 2007)

Added heartbeat & other improvements to scheduler. Added db initialization to all agents.

0.5.0 (complete, 28 Nov 2007)

Lots of bugfixes

0.6.0 (complete, 18 Dec 2007)

First public release of FOSSology

0.6.1 (complete, 13 Feb 2008)

Added delagent to improve UI response times, added check for corrupt bsam file during install, tweaks to vacuum and analyze to try to eliminate SQL timeouts, major bSAM debugging

0.7.0 (complete, 18 Apr 2008)

Completely new web-based GUI, Modular, pluggable GUI, New “Getting Started” page, updates to cp2foss tools, delete/reset/resumbit/reprioritize jobs, search feature, initialization plugin, user administration feature, bug fixes, performance improvements

0.8.0 (complete, 27 May 2008)

Release Target: 15 May 2008

0.9.0 (complete, 3 July 2008)

Release Target: 24 June 2008

1.0.0 (complete, 17 December 2008)

Release Target: 30 October 2008

1.1.0 (complete, 17 July 2009)

Release Target: June 2009

1.2.0 (complete, July 2010)

Release Target: March 2010

1.3.0 (complete, January 19, 2011)

Release Target: December 2010

  • Groups. Implementing user groups inside of fossology is an indirect but critical requirement for 1.3 because tagging is dependent on having groups to administer tag permissions.
  • File tagging
    1. The ability to attach a tag (short (max 32 character) tag, plus a long text) to a file or container.
    2. Tagging requires permission for creating, attaching, and viewing tags. Access control at this level does not currently exist in fossology which is why we have a dependency on groups and a permission strategy.
  • Replace the copyright agent. A quick experiment showed that we can get better results with simple heuristics rather then the current naive Bayes. Development is on branches/new_copyright.
  • fix cp2foss authentication error that prevents bucket agent from getting scheduled.
  • Increase software Quality. This task involves a number of subactivites.
    1. C code Unit Test and Coverage suite. Initial proposal, ideas or framework for how to use C code unit tests and C code coverage to improve our code quality.
    2. PHPUnit Test Framework Investigate the feasibility of using this suite of tools forunit testing PHP, performing code coverage analysis of php code and test automation.
    3. Code Coverage for C code Investigate and produce a process that allows the capture of code coverage data for our C code using LCOV and GCOV.
  • Separate text in UI to facilitate i18N for Stefan.
  • Scheduler.conf does not get created correctly for a cluster. fosscp_agent and fo_notify should only be run on the scheduler host (localhost). Don't forget to include an entry for selftest, too.
  • NFS I/O performance investigate and improve. NFS file I/O is the largest bottleneck for agents, so what can we do to make it faster? Besides faster, making repo access more robust, easier to admin, and take less disk space would be big improvements.
  • Remove copyrights from nomos scan. They are confusing since they also appear in the copyright scan, they slow down nomos (amount unknown), and removing them reduces the code we are maintaining which will make refactoring easier in the future.
  • Add/Modify licenses on-the-fly this would include a new permission level so that user can correct license and bucket data (other data as well). Perhaps this would update the real data record, but an audit trail could be kept of any changes. (reqt from sutula)
  • Improve the unpack agent. The unpack agent used by fossology extracts files from containers. A container is any kind of file that stores other files. For example, a ZIP file contains an archive of different files. Other types of containers include tar, ar, ISO, and rpm files. Look [[1.0.0:agents#unpack | here]] for a full description of unpack agent. What’s wrong with the current unpack agent?:
    1. The agent is SLOW. It can take days to unpack a Linux distro. Since Linux distros are of primary interest to the OSRB, fossology needs to be able to unpack distros in hours or minutes, not days. How can we take advantage of multiple CPUs (with the –m switch?) and agent systems to improve performance. DONE: Unpack performance.
    2. The current unpack agent cannot process some Microsoft proprietary formatted files. Look into how we do this on a Linux server. DONE unpack_microsoft_proprietary_formatted_files
    3. Since p7zip supports multiple threads and unpacks more than just microsoft install files, look into if it makes sense to replace some of the other utilities we use. * pending * replace some of the other utilities used
      1. The first step to this is to look at our install dependencies for ununpack. Then from that list, which could we remove if we used p7zip (since we already have a dependency on p7zip). So start by making a list of the ununpack dependencies.
    4. Are there libraries we can use instead the fork/exec'ing each unpacking utility? For all the commands we use, using libraries instead of the executables is something that could potentially, significantly, speed up ununpack. instead fork/exec with libraries * pending *
  • Deprecate (bSAM) license analyzer and licterms. Either remove entirely or move to its own, unsupported, package.
  • Display License changes by package version (moved from 1.2)
  • Display license differences on a per file basis between versions of any archive (rpm, tar, etc) (moved from 1.2). This includes Distro reports
  • Reporting, browsing by collections. (requested: SKP)
    1. A collection could be all the uploads in a folder, or it could be a user selected set of uploadtrees. To impliment do an sql union of the collection, then proceed normally. This applies to all browsers (browse, nomos, buckets, copyright).
    2. Reports on file lists (from Raino Sep 1, 2010). Reports (currently nomos, buckets, copyrights) should work on a file list as well as a container and file. For example, it would be interesting to see what copyrights are in 'Misc-copyright' files or 'No license found'. The best way to do it would be to select first from Nomos License Browser 'Show Misc-copyright' and then Browse copyrights of those files only. Furthermore, this feature could be elaborated for Buckets as well. If one has bucket 'licenses that need further study' then probably she is most interested of copyrights of files in that bucket.

1.4 (completed, May 19, 2011)

Release Target: March 31, 2011

  • Diff Tool (Bob)
  • UI simplification (Mark & Mary to define)
    1. remove deprecated bsam from UI
    2. Work on inconsistencies and ease of use.
    3. The way you queue a job that has already been unpacked is different depending on if it is a new scan or a rescan. Of course, most rescan's don't work, but that's an issue that needs to be handled by the new modular agent/plugin design.
    4. Micromenu can get very cluttered. Look at replacing with a pull down menu whose contents will be determined by active plugins.
    5. Search should be an option at any browse level
  • Implement authentication using Siteminder (Vincent)
  • Improved user documentation (how to use FO training) (Mary)
  • Setup a new internal FO assessment system (Mary)
  • Install a T-o-T FO instance on fo.org for early feedback from FO community (pending discussion on FO email)
  • Bug fixes for 1.3.0
    • Bug 487 : upload_userid has datatype text instead of integer in the upload table.
    • Bug 575, DB error checking * fixed *
    • Bug 579, can not unpack rar files, release note in v1.3.0? * fixed? *
    • Bug 577, can not unpack pdf files, fixed, not merge to trunk * fixed *
    • Bug 578, can not unpack ext2, ext3, fat and ntfs image, * fixed *
    • Bug 581, can not unpack boot partitions, * fixed *
    • Bug 583, can not find appropriate command for unpacking ext2, ext3, fat, ntfs image and boot partitions, * fixed *
    • Bug 584, for the function ExtractDisk(), return value is wrong when the test case is exceptional, * fixed *
    • Bug 585, or the function ExtractISO(), return value is wrong when the test case is exceptional, * fixed *
    • Bug 595, please enhance UI option "Upload from URL" to allow multiple files, * fixed *
      items below are not critical and are somewhat complicated, will include in v1.4.1 (or a later release) as time allows
      ----
    • Bug 580, unpacked result are abnormal for some iso files
    • Bug 582, unpacked result are abnormal for some rpm packages
    • Bug 594, Information/error messages are unhelpful

1.4.1 (completed, July 1, 2011)

Release Target: June 2011

  1. Primarily a bug fix release.
  2. Improve speed of license list and license list download
  3. List files for bucket
    1. add option show and download file list
    2. add filter to remove containers
  4. List file for license - add option show and download file list
  5. New type of bucket object. Add "archive" to file and package. Archives are tar, bz2, zip, ...
  6. Upload from repository (svn, git)
  7. Refactor Nomos
  8. Allow users to fix copyright false positives.

2.0 (completed, TBD)

Release Target: December 2011

  1. New scheduler
  2. Changes to API
  3. Improved multihost configuration and installation.
  4. Modular plugins and agents
  5. UI for bucket definition and management (new, change, delete)
  6. Information/error messages are unhelpful, non-existent and difficult to find in the log file. Log meaningful messages with names of file being processed (if applicable) to a log file for a specific upload – NOT the general fossology.log file. Perfect the message and log organization
  7. Add/Modify licenses on-the-fly this would include a new permission level so that user can correct license and bucket data (other data as well). Perhaps this would update the real data record, but an audit trail could be kept of any changes. (reqt from sutula) This should also be done to correct copyright results (raino)
  8. Improve the unpack agent. The unpack agent used by fossology extracts files from containers. A container is any kind of file that stores other files. For example, a ZIP file contains an archive of different files. Other types of containers include tar, ar, ISO, and rpm files. Look [[1.0.0:agents#unpack | here]] for a full description of unpack agent. What’s wrong with the current unpack agent?:
    1. The agent is SLOW. It can take days to unpack a Linux distro. Since Linux distros are of primary interest to the OSRB, fossology needs to be able to unpack distros in hours or minutes, not days. How can we take advantage of multiple CPUs (with the –m switch?) and agent systems to improve performance. DONE: Unpack performance.
    2. The current unpack agent cannot process some Microsoft proprietary formatted files. Look into how we do this on a Linux server. DONE unpack_microsoft_proprietary_formatted_files
    3. Since p7zip supports multiple threads and unpacks more than just microsoft install files, look into if it makes sense to replace some of the other utilities we use. * pending * replace some of the other utilities used
      1. The first step to this is to look at our install dependencies for ununpack. Then from that list, which could we remove if we used p7zip (since we already have a dependency on p7zip). So start by making a list of the ununpack dependencies.
    4. Are there libraries we can use instead the fork/exec'ing each unpacking utility? For all the commands we use, using libraries instead of the executables is something that could potentially, significantly, speed up ununpack. instead fork/exec with libraries * pending *
  9. Deprecate (bSAM) license analyzer and licterms. Either remove entirely or move to its own, unsupported, package.
  10. Information/error messages are unhelpful, non-existent and difficult to find in the log file. Log meaningful messages with names of file being processed (if applicable) to a log file for a specific upload – NOT the general fossology.log file. Pending Perfect the message and log organization
  11. UI for bucket definition and management (new, change, delete) Not sure where this goes in the priorities.
  12. UI cleanup. Work on inconsistencies and ease of use. Some problems are:
    1. The way you queue a job that has already been unpacked is different depending on if it is a new scan or a rescan. Of course, most rescan's don't work, but that's an issue that needs to be handled by the new modular agent/plugin design.
    2. Micromenu can get very cluttered. Look at replacing with a pull down menu whose contents will be determined by active plugins.
    3. Search should be an option at any browse level.
  13. Display license changes by package version (moved from 1.2)
  14. Display license differences on a per file basis between versions of any archive (rpm, tar, etc) (moved from 1.2). This includes Distro reports
  15. Reporting, browsing by collections. (requested: SKP)
    1. A collection could be all the uploads in a folder, or it could be a user selected set of uploadtrees. To impliment do an sql union of the collection, then proceed normally. This applies to all browsers (browse, nomos, buckets, copyright).
    2. Reports on file lists (from Raino Sep 1, 2010). Reports (currently nomos, buckets, copyrights) should work on a file list as well as a container and file. For example, it would be interested to see what are copyrights in 'Misc-copyright' files or 'No license found'. The best way to do it would be to select first from Nomos License Browser 'Show Misc-copyright' and then Browse copyrights of those files only. Furthermore, this feature could be elaborated for Buckets as well. If one has bucket 'licenses that need further study' then probably she is most interested of copyrights of files in that bucket.
  16. Identify binary packages and the source package they came from (Scott Lamons). The issue here is that the source may not be in the same upload as the binary. So when looking at a binary we need to have an option to choose a source and look at its scans.

Future

  • We have a Roadmap page of things that would be interesting to do with FOSSology.

(See also How to create a new release for more info on release numbering and other useful info)