Add exec statistics processor & derby database

[jabaws.git] / TODO.txt

TODO: \r
\r
Current stat collector has to rely on file dates! \r
\r
Test all WS as names from Executables were removed\r
\r
Add default names for input and output in every executable and use them consistently\r
throughts (e.g. in all WS). Best of all use the same name from SceletalExectuable \r
For statistics. \r
\r
FIXME: \r
Conecting to JABAWS version 2 service\r
09-Feb-2011 15:27:53 compbio.ws.client.Jws2Client connect\r
INFO: Connected successfully!\r
Exception in thread "main" java.lang.NullPointerException\r
        at compbio.ws.client.MetadataHelper.getParametersList(MetadataHelper.java:30)\r
        at compbio.ws.client.Jws2Client.<init>(Jws2Client.java:179)\r
        at compbio.ws.client.Jws2Client.main(Jws2Client.java:483)\r
        \r
ScoreManager should output scores properly \r
\r
Allow empty Parameters and Preset files! \r
\r
Check the WS input and reject it on submission rather then of access with error message\r
\r
Globprot need a proper reference to bio python and sav_gol binaries -> they should be \r
somehow taken from disembl. \r
\r
Add registry service to query services status\r
\r
Refactor web services checker to enable a programmatic access to its methods.\r
Rename it to avoid confusion with jabaws client\r
\r
Finish the client\r
\r
Add interface for Jalview annotation \r
Add the method to return Jalview Annotation to SequenceAnnotation IF  \r
\r
Develop generic Interface to return Jalview annotation for easy to add new \r
services (?) \r
\r
Replace conservation.Method with server.ws.Method \r
and try building WS. If this does not work - get rid of Method\r
\r
Output file parsing for stat reporting\r
cluster engine stat of www-jws2 user\r
\r
integrate the above to tweak the size of the local job\r
\r
Add AACon ws\r
Add iupred ws http://iupred.enzim.hu/\r
Add globprot ws - does not report raw scores, just regions\r
Add ronn ws\r
\r
Philogeny Mrbayes + Philip\r
 \r
USE CASE - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE\r
- Receive user alignment \r
- use hmmerbuild to turn it to profile\r
- use hmmersearch to search the database \r
\r
#END OF - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE \r
\r
New data model for representing psiblast,blast,phmmer,jackhmmer results\r
\r
new parsers for the above programmes output (Stockholm MSA format?)\r
\r
Think hard on what to do with large output files? \r
e.g. serve the hits table in full, but retrieve alignments on demand.\r
What actually needs to be sent?   \r
\r
Add facility to distribute other results of the calculations like the trees and \r
annotation file for probcons. \r
\r
# END OF SEARCHING SEQUENCE DATABASE USECASE\r
\r
Good toString method for Limits (test with command line client -limits)\r
\r
JABA DOCS\r
(later) Define limits for presets - if required\r
(later) Add documentation on Presets, Parameters and Limits\r
(later) Explain how to define a limit\r
\r
Add test for repeated result collection from cluster engine\r
\r
(low) Pack the test cases and build file to run them in one of the distributives \r
\r
(low) Add logging facility to WSTester so more details of the problem can be reported\r
\r
(low) Statistics collector for engines (number of operations/timing)\r
\r
(later) WRAP Amps\r
\r
(later) Implement utility to rerun died tasks\r
\r
(later) Add recognition for executables for the different architectures\r
\r
(later) use Latch to manage engine reservations. Reservations for parallel jobs? \r
\r
DONE\r
WILL NOT DO: Progress bars - needs assessing how long does it take to run a task\r
DONE: LocalExecutor service must be shutdown on web application undeploy event not on JVM shutdown. \r
      This is because JVM is still running even after web application is shut down!\r
DONE: Use absolute path for web site download links as archives are not included into distro!\r
DONE: Rearrange web site docs - add links to the top of the page to the topics discussed below\r
DONE: Put the documentation for various executables online\r
DONE: Make left gradient image one-two characters larger + change "For developer ->For Developers"\r
DONE: test local/cluster execution - test Load balancer\r
DONE: Improve SimpleWS client so it can be scripted against: give user alignment and accept parameters\r
DONE: Make a configuration tester class (check that configuration files point to the executables, and they can be executed)\r
DONE: PUT CLIENT JAR INTO WEB-INF/LIB - most likely need to get rid of dist prefix in the build file\r
DONE: Different packages for download and dundee. generic vs specific settings. \r
DONE: JAVADOC\r
DONE: Refactor EngineResourcesLeak tester to hide a map implementation!\r
DONE: Compile executables for linux in the most generic way\r
DONE: Make different log4j configuration files / web services logs.\r
DONE: System manual\r
DONE: Developer doc\r
DONE: How to\r
DONE: Prepare different packages\r
DONE: extract commonly used names into variable in the build script\r
DONE: Test for high load conditions (TestNG?)\r
DONE: (low) Fix: Started jobs with unknown completion status reported as FINISHED by  msaws.getJobStatus(taskId) (see SimpleWSClient)\r
DONE: fix compile binaries script\r
DONE: Make script to change executable flag for linux systems\r
DONE: jobsout must be created within war\r
DONE: enable relative paths for local engine conf temp dir\r
COUND NOT REPRODUCE: fix cluster job status - RUNNING -> UNDEFINED\r
DONE: throw JobSubmission exception with appropriate method instead - (low) code discoverer/tester for ws - e.g. after war deployment to find out what actually works\r
DONE: fix build scripts after reorganising project\r
DONE: package as war file for deployment\r
DONE: package for jalview distro for local execution\r
DONE: reorganise project into independent chunks\r
DONE: test Parameter & Option toCommand method\r
DONE: Replace WS methods with new CommandBuilder.toCommand()\r
DONE: Setting default values is validated as external method \r
DONE: Refactoring to extract data model, and engines is performed\r
DONE: Default value is adopted for passing new Option values to server side\r
DONE: Converting Options to commands delegated to CommandBuilder  \r
DONE: Make Arguments mutable to facilitate default value setting for WS\r
DONE: Matrix files for alignment programme\r
DONE: Clean up testcases\r
DONE: Replace List<String> as command line parameters abstraction to dedicated CommandBuilder class \r
DONE: Parallel execution for tcoffee jobs\r
DONE: Copyright added\r
DONE: WRAP ProbCons\r
DONE: think about time limits and cluster parameters for parallel execution - NativeSpecs for cluster is implemented\r
DONE: (low) Use for WS: (@WebParam(name = "name") String name) {\r
DONE: Implement rough policy for runtimes (e.g. do not run excessively long tasks on the local machine if cluster is available)\r
DONE: Write tomcat conf to expose WS\r
DONE: access control to web.xml\r
DONE: Statistics collector for WS\r
DONE: implements limits for web services\r
DONE: WS clients must be able to connect to a web service given the service url. \r
e.i. ws URL should not be compiled in using wsimport, but determined from \r
the configuration file.  \r
\r
DONE: Could not find configuration files if run from within servlet container\r
DONE: Add ways to get engine based on the key for everything but submission of the new tasks \r
DONE: Add cache to Cluster JOBID - so that at least jobs for which results have not been collected is cached.\r
DONE: FINISHED overrides FINISHED with time stamp!\r
DONE: Consider think about JobStatus and StatFileStatus\r
DONE: a method to read 3 chars at the beginning of the file to indicate a progress is added\r
DONE: ConfiguredExecutable<?> al = sexec.getResults(jobId); || Alignment ?\r
DONE: ClustalW - OUTput must be set !\r
DONE: Add accessor methods to get a DUMP of the process output \r
DONE: FilePuller should be cached to avoid excessive fs exercise\r
DONE: Test List of the processes (ensure no memory leaks)\r
PART DONE - On the client on shutdown remember the task list - server tasks could be loaded and executed from saved run configurations, client should decide for themselves what they want\r
DONE: Persist the task list to enable later result collection on the server side\r
DONE Wrap TCoffee\r
DONE: improve parameters handling - new schema + update for all parameters\r
DONE: Cluster jobs report submitting and result collecting/finishing time (for sync jobs only(!)) and cluster jobid in the corresponding files  \r
DONE: Local jobs report starting and finising times in files with corresponding names in the task directory  \r
DONE: Generalize getResult method for MSA runners\r
DONE: Create test suits\r
DONE: Convert junit tests to TestNG\r
DONE: Redirect output & error streams to the named files as some processed need this  \r
DONE: Extract generilyzed class with most of the method for individual executable to use\r
DONE: Refit wrappers to use the generalised class to avoid duplication  \r
DONE: Pass environment variable settings to the executable\r
DONE: Complete AsyncExecutors implementation\r
DONE: Look to unify methods for engines\r
DONE: Provide way around of generated directory setup for test cases(?)\r
DONE: Fix the test cases for local engine (need to set directory on executable explicitly)\r
DONE: Make log files for processes err and out for local execution\r
DONE: Trace where AsyncExcel called with null Executable and attempts to create a new directory repeatedly\r
DONE: directory generator\r
DONE: tests for cancel \r
DONE tests for getStatus\r
DONE  ws async\r
DONE: parametrized clustal operations\r
DONE: test large datasets handling \r
DONE: write load balancer \r
DONE: (low) task persistance\r
DONE: Look at CancellableTask if with Sync and AsyncExecutor \r