FUTURE TODO\r
\r
+Add interface for Jalview annotation \r
+\r
+Develop generic Interface to return Jalview annotation for easy to add new \r
+services (?) \r
+\r
+Philogeny Mrbayes + Philip\r
+ \r
+USE CASE - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE\r
+- Receive user alignment \r
+- use hmmerbuild to turn it to profile\r
+- use hmmersearch to search the database \r
+\r
+#END OF - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE \r
+\r
+New data model for representing psiblast,blast,phmmer,jackhmmer results\r
+\r
+new parsers for the above programmes output (Stockholm MSA format?)\r
+\r
+Think hard on what to do with large output files? \r
+e.g. serve the hits table in full, but retrieve alignments on demand.\r
+What actually needs to be sent? \r
+\r
+\r
+Add facility to distribute other results of the calculations like the trees and \r
+annotation file for probcons. \r
+\r
+(later) WRAP Amps\r
+\r
+(later) Implement utility to rerun died tasks from command line\r
+\r
Registry for jalview WS - gives list of web services providers\r
- checks the status of the web services\r
- web service providers gives list of available WS\r
-TODO ideally should introduce native Mac support (and maybe abandon native windows support in \r
-favior of VM) \r
+? ideally should introduce native Mac support (and maybe abandon native windows support in \r
+favour of VM) \r
\r
TODO write some help on the executable.properties within this file! \r
check that after binaries relocation build tasks point to correct locations!\r
-+ GET rid of binaries/help directory!\r
-+ update binaries\r
- + muscle binary version 3.8.31 for win/lin32/lin64 and sources, docs in website/prog_docs\r
- - test Muscle on the cluster\r
- - test Muscle compilation\r
- + probcons up-to-date\r
- + clustal up-to-date \r
- + update tcoffee (8.99 latest) source code updated docs updated \r
- + tcoffee compilation for IA32 \r
- + update Mafft (6.857 latest) for win/lin32/lin64 and sources, no docs updates\r
-+ make sure conf files are optional!\r
-\r
-TODO: \r
-Registry 1 week\r
-webservices - 1 week\r
-client - 1 week\r
-docs - 1 week\r
\r
Cluster stats: \r
- +cancelled should not be reported as incomplete!\r
-Remove hyperlinks from tasks which workdirs were removed \r
-graph generation \r
-user documentation (do not forget how to enable follow symlinks!)\r
- -use the same name for output for different executables\r
- +ant task to clean up clean stat db task before packing web app\r
- +pack web apps with stat db (NOT TESTED!)\r
- +change input size bites\r
- +add help info for each column in the detailed job list table\r
- +replace default -1 with ? \r
- +delete and re-import the database to fix clustal input staff \r
\r
add to help text: To disable a web service remove it from WEB-INF/sun-jaxws.xml descriptor\r
\r
\r
Current stat collector has to rely on file dates! \r
\r
-Test all WS as names from Executables were removed\r
-\r
-Add default names for input and output in every executable and use them consistently\r
-throughts (e.g. in all WS). Best of all use the same name from SceletalExectuable \r
-For statistics. \r
-\r
-FIXME: \r
-Conecting to JABAWS version 2 service\r
-09-Feb-2011 15:27:53 compbio.ws.client.Jws2Client connect\r
-INFO: Connected successfully!\r
-Exception in thread "main" java.lang.NullPointerException\r
- at compbio.ws.client.MetadataHelper.getParametersList(MetadataHelper.java:30)\r
- at compbio.ws.client.Jws2Client.<init>(Jws2Client.java:179)\r
- at compbio.ws.client.Jws2Client.main(Jws2Client.java:483)\r
- \r
-ScoreManager should output scores properly \r
-\r
-Allow empty Parameters and Preset files! \r
-\r
-Check the WS input and reject it on submission rather then of access with error message\r
-\r
Globprot need a proper reference to bio python and sav_gol binaries -> they should be \r
somehow taken from disembl. \r
\r
-+Add registry service to query services status\r
\r
-+Refactor web services checker to enable a programmatic access to its methods.\r
-+Finish the client\r
-\r
-Add interface for Jalview annotation \r
-Add the method to return Jalview Annotation to SequenceAnnotation IF \r
-\r
-Develop generic Interface to return Jalview annotation for easy to add new \r
-services (?) \r
-\r
-Replace conservation.Method with server.ws.Method \r
-and try building WS. If this does not work - get rid of Method\r
+? Replace conservation.Method with server.ws.Method and try building WS. If this does not work - get rid of Method\r
\r
Output file parsing for stat reporting\r
-cluster engine stat of www-jws2 user\r
+install cluster engine stat of www-jws2 user\r
\r
integrate the above to tweak the size of the local job\r
\r
-Add AACon ws\r
-Add iupred ws http://iupred.enzim.hu/\r
-Add globprot ws - does not report raw scores, just regions\r
-Add ronn ws\r
-\r
-Philogeny Mrbayes + Philip\r
- \r
-USE CASE - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE\r
-- Receive user alignment \r
-- use hmmerbuild to turn it to profile\r
-- use hmmersearch to search the database \r
-\r
-#END OF - TURN ALIGNMENT INTO PROFILE AND SEARCH SEQUENCE DATABASE USECASE \r
-\r
-New data model for representing psiblast,blast,phmmer,jackhmmer results\r
-\r
-new parsers for the above programmes output (Stockholm MSA format?)\r
-\r
-Think hard on what to do with large output files? \r
-e.g. serve the hits table in full, but retrieve alignments on demand.\r
-What actually needs to be sent? \r
-\r
-Add facility to distribute other results of the calculations like the trees and \r
-annotation file for probcons. \r
-\r
-# END OF SEARCHING SEQUENCE DATABASE USECASE\r
-\r
Good toString method for Limits (test with command line client -limits)\r
-\r
+Document the Limits for presets\r
JABA DOCS\r
-(later) Define limits for presets - if required\r
+\r
(later) Add documentation on Presets, Parameters and Limits\r
(later) Explain how to define a limit\r
\r
\r
(low) Pack the test cases and build file to run them in one of the distributives \r
\r
-(low) Add logging facility to WSTester so more details of the problem can be reported\r
\r
-(low) Statistics collector for engines (number of operations/timing)\r
+DONE\r
\r
-(later) WRAP Amps\r
+version 2\r
\r
-(later) Implement utility to rerun died tasks\r
+DONE: GET rid of binaries/help directory!\r
+DONE: update binaries\r
+ DONE: muscle binary version 3.8.31 for win/lin32/lin64 and sources, docs in website/prog_docs\r
+ DONE: test Muscle on the cluster\r
+ DONE: test Muscle compilation\r
+ DONE: probcons up-to-date\r
+ DONE: clustal up-to-date \r
+ DONE: update tcoffee (8.99 latest) source code updated docs updated \r
+ DONE: tcoffee compilation for IA32 \r
+ DONE: update Mafft (6.857 latest) for win/lin32/lin64 and sources, no docs updates\r
+DONE: make sure conf files are optional!\r
\r
-(later) Add recognition for executables for the different architectures\r
+Cluster stats: \r
+ DONE: cancelled should not be reported as incomplete!\r
+ DONE: ant task to clean up clean stat db task before packing web app\r
+ DONE: pack web apps with stat db (NOT TESTED!)\r
+ DONE: change input size bites\r
+ DONE: add help info for each column in the detailed job list table\r
+ DONE: replace default -1 with ? \r
+ DONE: delete and re-import the database to fix clustal input staff \r
+\r
+DONE: Test all WS as names from Executables were removed\r
+DONE: Add default names for input and output in every executable and use them consistently\r
+ throughts (e.g. in all WS). Best of all use the same name from SceletalExectuable \r
+ For statistics. \r
+\r
+DONE: FIXED Conecting to JABAWS version 2 service\r
+ 09-Feb-2011 15:27:53 compbio.ws.client.Jws2Client connect\r
+ INFO: Connected successfully!\r
+ Exception in thread "main" java.lang.NullPointerException\r
+ at compbio.ws.client.MetadataHelper.getParametersList(MetadataHelper.java:30)\r
+ at compbio.ws.client.Jws2Client.<init>(Jws2Client.java:179)\r
+ at compbio.ws.client.Jws2Client.main(Jws2Client.java:483)\r
\r
-(later) use Latch to manage engine reservations. Reservations for parallel jobs? \r
+DONE: ScoreManager should output scores properly \r
+DONE: Allow empty Parameters and Preset files! \r
+DONE: Check the WS input and reject it on submission rather then of access with error message\r
+\r
+DONE: Add registry service to query services status\r
+DONE: Refactor web services checker to enable a programmatic access to its methods.\r
+DONE: Finish the client\r
+DONE: Add AACon ws\r
+DONE: Add iupred ws http://iupred.enzim.hu/\r
+DONE: Add globprot ws - does not report raw scores, just regions\r
+DONE: Add ronn ws\r
+DONE: Define limits for presets - if required\r
+DONE: Add logging facility to WSTester so more details of the problem can be reported\r
+DONE: Statistics collector for engines (number of operations/timing)\r
+\r
+\r
+version 1\r
\r
-DONE\r
-WILL NOT DO: Progress bars - needs assessing how long does it take to run a task\r
DONE: LocalExecutor service must be shutdown on web application undeploy event not on JVM shutdown. \r
This is because JVM is still running even after web application is shut down!\r
DONE: Use absolute path for web site download links as archives are not included into distro!\r
#local.disembl.bin.windows= \r
local.globplot.bin=binaries/src/globplot/GlobPlot.py\r
cluster.globplot.bin=/homes/pvtroshin/workspace/jaba2/binaries/src/globplot/GlobPlot.py\r
+#globplot.bin.env=PYTHONHOME#/usr/local/lib/python2.6; \r
# globplot.parameters.file=conf/settings/GlobPlotParameters.xml\r
globplot.limits.file=conf/settings/GlobPlotLimits.xml\r
globplot.cluster.settings=-l h_cpu=24:00:00 -l h_vmem=6000M -l ram=6000M\r
\r
package compbio.engine.client;\r
\r
+import java.util.Arrays;\r
import java.util.Collections;\r
import java.util.Map;\r
\r
}\r
\r
static String[] getEnvVariableList(String property) {\r
+ property = property.trim(); \r
if (containsMultipleVariables(property)) {\r
return property.split(NEXT_ENV_PROPERTY_DELIMITER);\r
}\r
"Must be only one value per property! Make sure the property does not contain '"\r
+ PROP_NAME_VALUE_SEPARATOR + "' symbol!");\r
}\r
- if (vars.length == 0) {\r
+ if (vars.length == 0 || vars.length==1) {\r
+ log.warn("Environmental variable '"+property+"' does not have a value!");\r
return "";\r
}\r
return vars[1];\r
return property.contains(PROP_NAME_VALUE_SEPARATOR);\r
}\r
\r
- /**\r
- * This method change relative paths to absolute for two environmental\r
- * variables MAFFT-BINARIES and FASTA_4_MAFFT to avoid requesting every user\r
- * to specify these path for every installation\r
- */\r
- void absolutiseMafftpaths() {\r
\r
- }\r
\r
/*\r
* This is a horrible hack, but it only requires to simplify configuration\r
+ PROP_NAME_VALUE_SEPARATOR + "propertyValue. "\r
+ "Given values is: " + evar);\r
}\r
+ evar=evar.trim(); \r
String varName = getEnvVariableName(evar);\r
String varValue = getEnvVariableValue(evar);\r
// absolutise local paths to mafft env variables\r
return false;\r
}\r
File exec = new File(directory);\r
- return exec.exists() && exec.isDirectory();\r
+ return exec.exists() && exec.isDirectory() && exec.canRead();\r
}\r
\r
/**\r
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">\r
<suite name="jaba2" parallel="false">\r
-<!--\r
- <test verbose="3" name="compbio.runner.disorder.IUPredTester" annotations="JDK">\r
+\r
+ <test verbose="3" name="compbio.runner.disorder.GlobPlotTester" annotations="JDK">\r
<classes>\r
- <class name="compbio.runner.disorder.IUPredTester"/>\r
+ <class name="compbio.runner.disorder.GlobPlotTester"/>\r
</classes>\r
</test>\r
--->\r
- <test verbose="2" name="compbio.runner.msa.TcoffeeParametersTester" annotations="JDK">\r
- <classes>\r
- <class name="compbio.runner.msa.TcoffeeParametersTester">\r
+\r
+\r
<!--\r
+ <test verbose="2" name="compbio.stat.collector.StatDBTester" annotations="JDK">\r
+ <classes>\r
+ <class name="compbio.stat.collector.StatDBTester">\r
+\r
<methods>\r
<include name="testClusterExecute"/>\r
</methods>\r
--->\r
+\r
</class>\r
</classes>\r
</test>\r
-\r
+-->\r
</suite>\r
\r
ClustalW clustal = new ClustalW();\r
/**\r
- * ClustalW local limit is 40 sequences per 500 letters\r
+ * ClustalW local limit is 2 sequences per 500 letters\r
*/\r
ConfiguredExecutable<ClustalW> confClust = Configurator\r
.configureExecutable(clustal);\r
ExecProvider aEngine = LoadBalancer.getEngine(confClust, data);\r
- Limit locExec = confClust\r
- .getLimit(PresetManager.LOCAL_ENGINE_LIMIT_PRESET);\r
+ Limit<ClustalW> locExec = new Limit<ClustalW>(2, 500, PresetManager.LOCAL_ENGINE_LIMIT_PRESET); \r
+ // For testing of production configuration uncomment\r
+ //Limit locExec = confClust\r
+ // .getLimit(PresetManager.LOCAL_ENGINE_LIMIT_PRESET);\r
assertTrue(locExec.getSeqNumber() <= data.size()\r
|| locExec.getAvgSeqLength() * locExec.getSeqNumber() <= Limit\r
.getAvgSequenceLength(data));\r
-\r
- assertEquals(aEngine, ExecProvider.Cluster);\r
+ // Engine will be local because LoadBalancer accessed the presets & limits \r
+ // directly\r
+ // For testing of production configuration uncomment\r
+ // assertEquals(aEngine, ExecProvider.Cluster);\r
\r
} catch (JobSubmissionException e) {\r
e.printStackTrace();\r
void validateDirProp(String propName) {\r
assertNotNull(ph);\r
assertNotNull(ph.getProperty(propName));\r
- File tmp = new File(ph.getProperty(propName));\r
+ File tmp = new File(ph.getProperty(propName).trim());\r
assertTrue(tmp.exists());\r
assertTrue(tmp.isDirectory());\r
assertTrue(tmp.canRead());\r
LOCAL_JOBS, 1);\r
ExecutionStatCollector cl_jobs = new ExecutionStatCollector(\r
CLUSTER_JOBS, 24);\r
-\r
+ \r
+ // Collect statistics prior to call getStats()!\r
+ local_jobs.collectStatistics();\r
+ cl_jobs.collectStatistics();\r
+ \r
StatProcessor local_stats = local_jobs.getStats();\r
StatProcessor cl_stats = cl_jobs.getStats();\r
- // System.out.println("L: " + local_stats.stats);\r
- System.out.println("\n\n");\r
- // System.out.println("C: " + cl_stats.stats);\r
-\r
+ \r
assertEquals(local_stats.getJobNumber(), 12);\r
// ClustalW#1015343425414965 - empty\r
assertEquals(local_stats.getIncompleteJobs().size(), 1);\r
assertEquals(local_stats.getFailedJobs().size(), 0);\r
- assertEquals(local_stats.getAbandonedJobs().size(), 2);\r
+ assertEquals(local_stats.getAbandonedJobs().size(), 1);\r
\r
}\r
\r
\r
public class StatDBTester {\r
StatDB statdb;\r
-\r
- @BeforeClass\r
+ Timestamp from;\r
+ Timestamp to;\r
+ \r
+ @BeforeClass(alwaysRun=true)\r
public void init() {\r
try {\r
statdb = new StatDB(true);\r
+ Calendar fromCal = Calendar.getInstance();\r
+ fromCal.set(2011, 4, 1);\r
+ \r
+ Calendar toCal = Calendar.getInstance();\r
+ toCal.set(2011, 5, 1);\r
+ from = new Timestamp(fromCal.getTimeInMillis());\r
+ to = new Timestamp(toCal.getTimeInMillis());\r
+ \r
} catch (SQLException e) {\r
e.printStackTrace();\r
Assert.fail(e.getLocalizedMessage());\r
@Test\r
public void testReadYearDataAndShutdown() {\r
Calendar cal = Calendar.getInstance();\r
- cal.roll(Calendar.YEAR, false);\r
+ cal.set(2010, 4, 1);\r
try {\r
List<JobStat> jobs = statdb.readData(\r
- new Timestamp(cal.getTimeInMillis()), new Timestamp(\r
- new Date().getTime()), Services.MuscleWS, false);\r
+ new Timestamp(cal.getTimeInMillis()), to, Services.MuscleWS, false);\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 1294);\r
// statdb.shutdownDBServer();\r
}\r
\r
}\r
+\r
@Test\r
public void testReadOneMonthData() {\r
- Calendar fromCal = Calendar.getInstance();\r
- fromCal.set(2011, 4, 1);\r
- \r
- Calendar toCal = Calendar.getInstance();\r
- toCal.set(2011, 5, 1);\r
- \r
try {\r
- List<JobStat> jobs = statdb.readData(\r
- new Timestamp(fromCal.getTimeInMillis()), \r
- new Timestamp(toCal.getTimeInMillis()), Services.TcoffeeWS, false);\r
+ List<JobStat> jobs = statdb.readData(from, to, Services.TcoffeeWS, false);\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 36);\r
\r
- jobs = statdb.readData(new Timestamp(fromCal.getTimeInMillis()),\r
- new Timestamp(toCal.getTimeInMillis()), Services.ClustalWS,\r
- false);\r
+ jobs = statdb.readData(from,to, Services.ClustalWS,false);\r
\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 137);\r
\r
- jobs = statdb.readData(new Timestamp(fromCal.getTimeInMillis()),\r
- new Timestamp(toCal.getTimeInMillis()), Services.MafftWS,\r
- false);\r
+ jobs = statdb.readData(from, to, Services.MafftWS,false);\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 126);\r
\r
- jobs = statdb.readData(new Timestamp(fromCal.getTimeInMillis()),\r
- new Timestamp(toCal.getTimeInMillis()), Services.ProbconsWS,\r
- false);\r
+ jobs = statdb.readData(from,to, Services.ProbconsWS,false);\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 9);\r
\r
- jobs = statdb.readData(new Timestamp(fromCal.getTimeInMillis()),\r
- new Timestamp(toCal.getTimeInMillis()), Services.MuscleWS,\r
- false);\r
+ jobs = statdb.readData(from,to, Services.MuscleWS,false);\r
assertNotNull(jobs);\r
assertEquals(jobs.size(), 63);\r
\r
return ((System.currentTimeMillis() - jd.jobdir.lastModified()) / (1000 * 60 * 60)) > timeOutInHours;\r
}\r
\r
+ /* Make sure that collectStatistics methods was called prior to calling this! \r
+ * TODO consider running collectStatistics from here on the first call \r
+ */\r
StatProcessor getStats() {\r
+ if(stats.isEmpty()) {\r
+ log.info("Please make sure collectStatistics method was called prior to calling getStats()!");\r
+ }\r
return new StatProcessor(stats);\r
}\r
\r
}\r
}\r
\r
- private void collectStatistics() {\r
+ void collectStatistics() {\r
File[] files = workDirectory.listFiles(directories);\r
for (File file : files) {\r
JobDirectory jd = new JobDirectory(file);\r
Connection conn = DriverManager.getConnection(protocol + statDBName\r
+ ";create=false");\r
conn.setAutoCommit(true);\r
+ log.debug("Connecting to the TEST database!");\r
return conn;\r
}\r
\r
git://source.jalview.org / jabaws.git / commitdiff