website/man_server_dev.jsp

   1 <%--<?xml version="1.0" encoding="ISO-8859-1" ?>--%>
   2 <%--<%@ page language="java" contentType="text/html; charset=ISO-8859-1" pageEncoding="ISO-8859-1"%>--%>
   3
   4 <%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
   5 <%@ taglib uri="http://java.sun.com/jsp/jstl/functions" prefix="fn" %>
   6 <%@ taglib uri="http://java.sun.com/jsp/jstl/fmt" prefix="fmt" %>
   7 <%@ taglib uri="http://displaytag.sf.net" prefix="dt" %>
   8
   9
  10 <c:import url="template_header.jsp" >
  11         <c:param name="title">Documentation</c:param>
  12 </c:import>
  13
  14 <ol class="breadcrumb">
  15         <li><a href="${pageContext.request.contextPath}/index.jsp">Home</a></li>
  16         <li><a href="man_docs.jsp">Documentation</a></li>
  17         <li class="active"><a href="man_server_dev.jsp">Developing JABAWS</a></li>
  18 </ol>
  19 <div class="row">
  20         <div class="col-md-12">
  21                 <div class="panel panel-default">
  22                         <div class="panel panel-heading">
  23                                 <h1 class="panel-title">For JABAWS Developers</h1>
  24                         </div>
  25                         <div class="panel-body">
  26                                 <!--<h4>JABAWS Server Virtual Appliance</h4>-->
  27                                 <ul>
  28                                         <li><a href="#api">The API</a></li>
  29                                         <li><a href="#thesrc">The source code</a></li>
  30                                         <li><a href="#projstruct">Structure of the project</a></li>
  31                                         <li><a href="#thecode">The code structure</a></li>
  32                                         <li><a href="#runtests">Running tests</a></li>
  33                                         <li><a href="#prepexec">Preparing distributive's</a></li>
  34                                         <li><a href="#buildstubs">Building web services artifacts</a></li>
  35                                         <li><a href="#addnewws">Adding a new web service (rough guide)</a></li>
  36                                 </ul>
  37                                 <!--<p class="justify">-->
  38                                 <!--</p>-->
  39                         </div>
  40                 </div>
  41         </div>
  42 </div>
  43 <div class="row" id="api">
  44         <div class="col-md-12">
  45                 <div class="panel panel-default">
  46                         <div class="panel panel-heading">
  47                                 <h1 class="panel-title">The API</h1>
  48                         </div>
  49                         <div class="panel-body">
  50                                 <p>This page is for anyone who wants to add a new webservice into the JABAWS framework.</p>
  51                                 <p><a href="${pageContext.request.contextPath}/dm_javadoc/index.html">Data Model JavaDoc</a> - read this if your are coding
  52                                         against JABA Web Services
  53                                 </p>
  54                                 <p><a href="${pageContext.request.contextPath}/full_javadoc/index.html">Complete JavaDoc</a> - for developers who want to
  55                                         use JABAWS framework and use Engines and Executables directly
  56                                 </p>
  57                                 <p class="text-right">
  58                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
  59                                 </p>
  60                         </div>
  61                 </div>
  62         </div>
  63 </div>
  64 <div class="row" id="thesrc">
  65         <div class="col-md-12">
  66                 <div class="panel panel-default">
  67                         <div class="panel panel-heading">
  68                                 <h1 class="panel-title">The source code</h1>
  69                         </div>
  70                         <div class="panel-body">
  71                                 <p><strong>Publicly available Git repository:</strong>
  72                                         <a href="http://source.jalview.org/gitweb/?p=jabaws.git;a=summary">
  73                                                 http://source.jalview.org/gitweb/?p=jabaws.git;a=summary</a></p>
  74                                 <p>Another publicly available JABAWS repository, containing the code for each
  75                                         JABAWS public release, is located at
  76                                         <a href="https://jabaws.googlecode.com">jabaws.googlecode.com</a></p>
  77                                 <p>The repositories contain a complete JABAWS
  78                                         <a href="http://www.eclipse.org">Eclipse</a> project. </p>
  79                                 <p>Of course if you want to make a modification to the source code you would need to
  80                                         generate distributives yourself. To do that, first generate JAX-WS artifacts using
  81                                         <em2>build-server</em2> task from <em2>wsbuild.xml</em2>
  82                                         ant script, then you could use build.xml tasks to generate any of the distributives you need.
  83                                 </p>
  84                                 <p class="text-right">
  85                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
  86                                 </p>
  87                         </div>
  88                 </div>
  89         </div>
  90 </div>
  91 <div class="row" id="projstruct">
  92         <div class="col-md-12">
  93                 <div class="panel panel-default">
  94                         <div class="panel panel-heading">
  95                                 <h1 class="panel-title">Structure of the project</h1>
  96                         </div>
  97                         <div class="panel-body">
  98                                 <ul>
  99                                         <li><i class="fa fa-folder-o" aria-hidden="true"></i> <span class="directory">binaries</span>   contains native executables e.g. clustalw</li>
 100                                         <li style="list-style-position:inside"><i class="fa fa-folder-o" aria-hidden="true"></i> <span class="directory">src</span>     contains sources of native executables</li>
 101                                         <li style="list-style-position:inside"><i class="fa fa-folder-o" aria-hidden="true"></i> <span class="directory">windows</span> contains pre-compiled Windows binaries</li>
 102                                         <li style="list-style-position:inside"><i class="fa fa-folder-o" aria-hidden="true"></i> compilebin.sh - the script to complile binaries</li>
 103                                         <li style="list-style-position:inside"><i class="fa fa-folder-o" aria-hidden="true"></i> setexecflag.sh - the script to set executable flag for the binaries</li>
 104                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> conf</span>       contains JABAWS configuration files     </li>
 105                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> ExecutionStatistics</span>        the database for storing collected execution statistics </li>
 106                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> jobsout</span> a default folder for temporary job directories </li>
 107                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> statpages</span> the web pages for execution statistics display</li>
 108                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> WEB-INF</span> default </li>
 109                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> website</span> contains the JABAWS web pages </li>
 110                                         <li style="list-style-position:inside"><i class="fa fa-folder-o" aria-hidden="true"></i> <span class="directory">archive</span> contains JABAWS packages, the WAR and JAR files</li>
 111                                 </ul>
 112                                 <p class="text-right">
 113                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
 114                                 </p>
 115                         </div>
 116                 </div>
 117         </div>
 118 </div>
 119 <div class="row" id="thecode">
 120         <div class="col-md-12">
 121                 <div class="panel panel-default">
 122                         <div class="panel panel-heading">
 123                                 <h1 class="panel-title">The code structure</h1>
 124                         </div>
 125                         <div class="panel-body">
 126                                 <img src="static/img/ws-structure.png" alt="JABAWS Code Structure"
 127                                          width="260" height="300" align="left" style="padding-right: 2em;" />
 128                                 <ul>
 129                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> datamodel</span> contains the JABAWS datamodel </li>
 130                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> engine</span> contains the JABAWS engine - the code that abstract the execution environment and executes native binaries</li>
 131                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> runner</span> contains the JABAWS runners - thin wrappers for native binaries </li>
 132                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> webservices</span> contains the JABAWS SOAP web services </li>
 133                                         <li><span class="directory"><i class="fa fa-folder-o" aria-hidden="true"></i> testsrc</span> contains the JABAWS unit tests</li>
 134                                 </ul>
 135                                 <p class="justify">
 136                                         Each source folder depends on the upper folders for compilation. For example, the datamodel
 137                                         is the top level folder so it has no other dependencies on other JABAWS code.
 138                                         The Engine level depends on the datamodel to compile etc. The web services folder is the
 139                                         bottom layer and depends on all the other source code.
 140                                 </p>
 141                                 <p class="justify">
 142                                         So the JABAWS project is split into 4 layers. From bottom-up the first layer
 143                                         consists from the value classes used by all other layers of the hierarchy, in
 144                                         particular web services. So, to be able to use JABAWS one needs to have these
 145                                         classes. At the same time classes on this layer does not have any dependencies
 146                                         on the layers above.
 147                                 </p>
 148                                 <p class="justify">
 149                                         The second layer contains code for execution of the wrappers, which are the
 150                                         abstraction describing native executables. JABAWS can execute tasks locally
 151                                         that is on the same machine as JVM and on the cluster. Thus currently code on
 152                                         this layer contain two engines. This layer depends on the layer underneath, the
 153                                         data model layer, but is completely independent from the code above.
 154                                 </p>
 155                                 <p class="justify">
 156                                         The third layer consists of the wrappers for the native executables and classes to
 157                                         handle their configuration. It depends on the engines and the data model, but know
 158                                         nothing about the web services.
 159                                 </p>
 160                                 <p class="justify">
 161                                         Finally, the upper layer contains the web services, that depend on all the layers below.
 162                                 </p>
 163                                 <p class="justify">
 164                                         The layer isolation is archived though specially designed compilation task which is
 165                                         executed sequentially in several stages so that the first layer compiles before
 166                                         any other layers, second layer compiles after that and process continies before
 167                                         all the code is compiled.  Any violation of the layer boundaries results in the
 168                                         compilation failure. Use Ant &quot;Compile&quot; or &quot;Complile_with_debug&quot;
 169                                         tasks to perform the staged compilation.
 170                                 </p>
 171                                 <p class="justify">
 172                                         A client package contains only classes from data model layer and a simple web
 173                                         services client. Framework package is for anyone who want to use JABAWS framework
 174                                         for controlling native executables in local or cluster environments. Framework exclude
 175                                         the web services layer. Server package contains all the code.
 176                                 </p>
 177                                 <p class="text-right">
 178                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
 179                                 </p>
 180                         </div>
 181                 </div>
 182         </div>
 183 </div>
 184 <div class="row" id="runtests">
 185         <div class="col-md-12">
 186                 <div class="panel panel-default">
 187                         <div class="panel panel-heading">
 188                                 <h1 class="panel-title">Running tests</h1>
 189                         </div>
 190                         <div class="panel-body">
 191                                 <p class="justify">
 192                                         JABAWS uses TestNG framework for testing. The test results for the JABAWS
 193                                         package offered for download can be found at:
 194                                         <a href="http://www.compbio.dundee.ac.uk/user/www-jws2/tests/index.html">
 195                                                 Test Results</a><br/>
 196                                         JABAWS uses <a href="http://testng.org/doc/index.html">TestNG</a>
 197                                         for testing. There is a TestNG plugin available for Eclipse
 198                                         which has functionality similar to JUnit. However, no plugins are
 199                                         necessary to run the test cases, as testng jar is supplied
 200                                         with JABAWS together with an ant tasks to run the test cases.
 201                                 </p>
 202                                 <p class="justify">
 203                                         The best way to ensure that JABAWS framework is completely functional on your
 204                                         system is to run all test cases.
 205                                         Test cases tests all aspects of JABAWS functionality. Consequently, one need
 206                                         to have non windows operation system and support
 207                                         of the cluster to be able to run all tests. If your system does not support
 208                                         cluster, then you could run all test excluding those that depends on the cluster.
 209                                 </p>
 210
 211                                 <p>Several testing groups are supported:</p>
 212                                 <ul>
 213                                         <li>All tests ('Test')</li>
 214                                         <li>Cluster tests ('Run_cluster_dependent_test')</li>
 215                                         <li>Cluster independent tests ('All_cluster_independent_tests') </li>
 216                                         <li>Windows only tests ('All_cluster_independent_windows_only_tests') </li>
 217                                         <li>Performance and stability tests ('Long_tests') </li>
 218                                         <li>Re-run failed tests ('Rerun_failed_tests') </li>
 219                                         <li>Run custom test ('CustomTest') </li>
 220                                 </ul>
 221
 222                                 <p class="justify">
 223                                         To run the tests you need to download all sources from <a href="#thesrc">repository</a>.
 224                                         Once you have done that, enter into the command line mode, change directory to the
 225                                         project directory and type:
 226                                 </p>
 227                                 <pre><code class="bash">ant -f build.xml &lt;test group name&gt;</code></pre>
 228                                 <p class="justify">
 229                                         Make sure you have <a href="http://ant.apache.org/">Apache Ant</a>
 230                                         installed and path to ant executable is defined in your path environmental variable.
 231                                         Replace test group name with the one of the names given in the list above to run required group of tests e.g for running cluster only tests
 232                                         use the following
 233                                         command:</p>
 234                                 <pre><code class="bash">ant -f build.xml Run_cluster_dependent_test</code></pre>
 235                                 <p class="justify">
 236                                         If you work under Linux you could use a simple script from the root folder of repository called <em2>runtests.sh</em2>
 237                                         This script simply contains a collection of the test commands described above and paths to java home directory and an ant executable,
 238                                         which you can define once for your system and then reuse.
 239                                 </p>
 240                                 <p class="justify">
 241                                         A handy feature of TestNG is its ability to re-run failed tests. Failed test
 242                                         ant file is stored in
 243                                         <em2>test-output/testng-failed.xml</em2>. and is used in the ant task called
 244                                         <em2>Rerun_failed_tests</em2>. So re-running failed tests requires no more
 245                                         work than running any other test group and could be accomplished with the command:
 246                                 </p>
 247                                 <pre><code class="bash">ant -f build.xml Rerun_failed_tests</code></pre>
 248                                 <p class="justify">
 249                                         CustomTest runs the test defined in the project root directory file called
 250                                         <em2>temp-testng-customsuite.xml</em2>. This
 251                                         file is generated by TestNG plugin every time you run the test from Eclipse.
 252                                         Thus an easy way to run a test in a different environment is to run it from
 253                                         Eclipse first and then from ant using a custom test procedure.
 254                                 </p>
 255                                 <p class="justify">
 256                                         For cluster execution make sure that the property
 257                                         LD_LIBRARY_PATH defined in build.xml points to cluster engine LD
 258                                         libraries directory in your local system.
 259                                 </p>
 260                                 <p class="text-right">
 261                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
 262                                 </p>
 263                         </div>
 264                 </div>
 265         </div>
 266 </div>
 267 <div class="row" id="prepexec">
 268         <div class="col-md-12">
 269                 <div class="panel panel-default">
 270                         <div class="panel panel-heading">
 271                                 <h1 class="panel-title">Preparing distributives</h1>
 272                         </div>
 273                         <div class="panel-body">
 274                                 <p class="justify">
 275                                         There are a number of ant tasks aimed for preparing distributives for download.
 276                                         Currently a few types of JABAWS packages are offered:
 277                                 </p>
 278                                 <ol>
 279                                         <li>Client only (contains classes required to access JABA Web Services)</li>
 280                                         <li>Platform specific JABAWS (windows and other)</li>
 281                                         <li>JABAWS without binaries</li>
 282                                         <li>JABAWS framework </li>
 283                                 </ol>
 284                                 <p>Corresponding build task names are:</p>
 285                                 <ol>
 286                                         <li>min-jaba-client</li>
 287                                         <li>jaba-windows, jaba-complete</li>
 288                                         <li>jaba-no-binaries</li>
 289                                         <li>full-jaba-client</li>
 290                                 </ol>
 291                                 <p class="justify">
 292                                         The easiest way to build all distributives is to call
 293                                         <em2>build-all</em2> ant task.
 294                                         There are more tasks defined in build.xml than described here.
 295                                         They are mostly self explanatory.
 296                                 </p>
 297                                 <p class="justify">
 298                                         If you made any changes to the data model and
 299                                         would like to generate a complete JABAWS
 300                                         distro make sure you have rebuilt jaxws artifact as described below.
 301                                 </p>
 302                                 <p class="text-right">
 303                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
 304                                 </p>
 305                         </div>
 306                 </div>
 307         </div>
 308 </div>
 309 <div class="row" id="buildstubs">
 310         <div class="col-md-12">
 311                 <div class="panel panel-default">
 312                         <div class="panel panel-heading">
 313                                 <h1 class="panel-title">Building web services artifacts</h1>
 314                         </div>
 315                         <div class="panel-body">
 316                                 <p class="justify">
 317                                         Server side artifacts should be rebuild whenever the data model, meta model or MSA
 318                                         interface were changed.
 319                                         To do that run build-server task from wsbuild.xml ant build file. WSDL files will
 320                                         be generated in
 321                                         <em2>webservices/compbio/ws/server/resource</em2> directory. It is not necessary to
 322                                         edit them if any of the JABAWS clients are used. JABAWS are the standard JAX-WS web
 323                                         services, which are WS-I basic profile compatible.
 324                                 </p>
 325                                 <p class="text-right">
 326                                         <a href="#">Back to top <i class="fa fa-arrow-up" aria-hidden="true"></i></a>
 327                                 </p>
 328                         </div>
 329                 </div>
 330         </div>
 331 </div>
 332 <div class="row" id="addnewws">
 333         <div class="col-md-12">
 334                 <div class="panel panel-default">
 335                         <div class="panel panel-heading">
 336                                 <h1 class="panel-title">Adding a new web service (rough guide)</h1>
 337                         </div>
 338                         <div class="panel-body">
 339                                 <ol>
 340                                         <li>
 341                                                 <p>Add a new executable which you'd like to wrap as a JABAWS web service to the binaries folder.
 342                                                         If it has the source code and can be recompiled for different platforms include it under
 343                                                         <em2>binaries/src</em2>. Edit <em2>setexecutableflag.sh</em2>
 344                                                         and <em2>compilebin.sh</em2> scripts in <em2>binaries/src</em2>
 345                                                         accordingly.</p>
 346                                         </li><li>
 347                                         <p>Make sure that all the dependencies of the software being installed are satisfied.
 348                                                 If there are other binaries they should be included as well. Keep the dependent
 349                                                 binaries in a subfolder for the main executable. Update
 350                                                 <em2>compilebin.sh</em2> and <em2>setexecflag.sh</em2> scripts accordingly.</p>
 351                                 </li><li>
 352                                         <p>Make sure that the new executable does not have any hard links to its dependencies, e.g. is able to run from
 353                                                 any installation folder and does not contain any hard coded paths.</p>
 354                                 </li><li>
 355                                         <p>Describe executable in <em2>conf/Exectuable.properties</em2> file. The lowercase name of
 356                                                 the wrapper should be included
 357                                                 in the name of the property for example Clustal properties all include clustal as a
 358                                                 part of the name e.g. local.clustalw.bin.
 359                                                 The same property for MAFFT will be called local.mafft.bin. For more help please refer
 360                                                 to the Executable.properties file.
 361                                         </p>
 362                                 </li><li>
 363                                         <p>Describe the executable supported parameters in the
 364                                                 <em2>&lt;ExecutableName&gt;Parameters.xml</em2>, presets in the
 365                                                 <em2>&lt;ExecutableName&gt;Presets.xml</em2> and the execution limits in the
 366                                                 <em2>&lt;ExecutableName&gt;Limit.xml</em2>. By convention these files are stored
 367                                                 in <em2>conf/settings</em2>. All of these are optional. If the executable
 368                                                 does not support parameters you do not have to mention the <em2>XXXParameter.xml</em2> file
 369                                                 in the <em2>Executable.properties</em2> file at all. The same is true for
 370                                                 Presets and Limits.</p>
 371                                 </li><li>
 372                                         <p>Create a Java wrapper class for your executable. Create it within <em2>runner</em2> source directory.
 373                                                 Examples of other wrappers can be found in <em2>compbio.runner.msa</em2> or
 374                                                 in other <em2>compbio.runner.*</em2> packages. Wrapper should extend
 375                                                 <em2>SkeletalExecutable&lt;T&gt;</em2> and implement
 376                                                 <em2>PipedExecutable&lt;T&gt;</em2>
 377                                                 if you need to pass the input or collect the results from the standard in/out. Please see Mafft
 378                                                 code as example. Wrapper should expend <em2>SkeletalExecutable&lt;T&gt;</em2>
 379                                                 if input/output can be set as a parameter for an executable. Please see the ClustalW code as example.</p>
 380                                 </li><li>
 381                                         <p>Create a testcase suit for your wrapper in <em2>testsrc</em2> and run the test cases.</p>
 382                                 </li><li>
 383                                         <p>Create parser for the output files of your executable. Suggested location
 384                                                 compbio.data.sequence.SequenceUtil . </p>
 385                                 </li><li>
 386                                         <p>Test the parser.</p>
 387                                 </li><li>
 388                                         <p>Decide which web services interfaces your executable is going to match. For example
 389                                                 if the executable output can be represented as SequenceAnnotation then SequenceAnnotation
 390                                                 interface might be appropriate. For multiple sequence alignment an Msa interface should be used.</p>
 391                                 </li><li>
 392                                         <p>If you find a web interface that matches your returning data type, then
 393                                                 implement a web service which confirms to it within a webservices source folder. </p>
 394                                 </li><li>
 395                                         <p>Register web service in <em2>WEB-INF/web.xml</em2>
 396                                                 and <em2>WEB-INF/sun-jaxws.xml</em2>.</p>
 397                                 </li><li>
 398                                         <p>Add generated wsdl to wsbuild.xml ant script to generate the stubs.</p>
 399                                 </li><li>
 400                                         <p>Run build-server task in wsbuild file. Watch for errors. If the task fails that means
 401                                                 that JAXB cannot serialize some of your new data structures. Add appropriate annotations to your data types.
 402                                                 Also check that: </p>
 403                                         <ul>
 404                                                 <li>you do not have interfaces to serialize, since JAXB cannot serialize them</li>
 405                                                 <li>you have a default no args constructor (can be private if you do not need it)</li>
 406                                                 <li>JAXB cannot serialize Java Map class, use a custom data structure instead</li>
 407                                                 <li>Enum cannot be serialized as its abstract class (do not confuse with enum which is fine)</li>
 408                                                 <li> Fields serialization leaves a little more space for manoeuvre.
 409                                                         If you do this then you may accept<br/> and return interfaces, e.g. List, Map; abstract classes etc, from your methods
 410                                                 </li>
 411                                         </ul>
 412                                         <p>If you have the data on the server side, but nothing is coming through to the
 413                                                 client, this is a JAXB serialization problem. They tend to be very silent and
 414                                                 thus hard to debug. Check your data structure can be serialized!</p>
 415                                 </li><li>
 416                                         <p>Modify the client to work with your new web service. Update Services
 417                                                 enumeration to include new service and ensure that all the methods of this
 418                                                 enumeration take into account the new service. Update the client help text
 419                                                 (client_help.txt) and insert it into the Constraints class.</p>
 420                                 </li><li>
 421                                         <p>Test the web service with the client.</p>
 422                                 </li><li>
 423                                         <p>Test on the cluster.</p>
 424                                 </li>
 425                                 </ol>
 426                         </div>
 427                 </div>
 428         </div>
 429 </div>
 430
 431 <jsp:include page="template_footer.jsp" />