From: Ben Soares Date: Mon, 15 May 2023 13:32:34 +0000 (+0100) Subject: JAL-629 Help and code with threads X-Git-Url: http://source.jalview.org/gitweb/?a=commitdiff_plain;h=refs%2Fheads%2Ffeatures%2FJAL-629_with_threads_NOT_PASSING_TESTS;p=jalview.git JAL-629 Help and code with threads --- diff --git a/help/help/html/features/clarguments-advanced.html b/help/help/html/features/clarguments-advanced.html index e69de29..249307a 100644 --- a/help/help/html/features/clarguments-advanced.html +++ b/help/help/html/features/clarguments-advanced.html @@ -0,0 +1,54 @@ + + +Jalview Command Line Arguments: advanced usage + + +

Jalview Command Line Arguments: advanced usage

+ +

+ Jalview Command Line Arguments: summary +
+ Jalview Command Line Arguments: introduction +
+ Jalview Command Line Arguments: basic usage +
+ Jalview Command Line Arguments: advanced usage +
+ Jalview Command Line Arguments: argfiles +

+ +
+ + + + + + + +
+ Continue to Jalview Command Line Arguments: argfiles. +
+ Return to Jalview Command Line Arguments: summary. + + + + diff --git a/help/help/html/features/clarguments-argfiles.html b/help/help/html/features/clarguments-argfiles.html new file mode 100644 index 0000000..a527c21 --- /dev/null +++ b/help/help/html/features/clarguments-argfiles.html @@ -0,0 +1,52 @@ + + +Jalview Command Line Arguments: argfiles + + +

Jalview Command Line Arguments: argfiles

+ +

+ Jalview Command Line Arguments: summary +
+ Jalview Command Line Arguments: introduction +
+ Jalview Command Line Arguments: basic usage +
+ Jalview Command Line Arguments: advanced usage +
+ Jalview Command Line Arguments: argfiles +

+ +
+ + + + + + + +
+ Return to Jalview Command Line Arguments: summary. + + + + diff --git a/help/help/html/features/clarguments-basic.html b/help/help/html/features/clarguments-basic.html index 19406b4..b557f4f 100644 --- a/help/help/html/features/clarguments-basic.html +++ b/help/help/html/features/clarguments-basic.html @@ -44,6 +44,7 @@
  • Adding 3D structures
  • Outputting/converting alignment files and images
  • Filename substitutions and batch processing
    • +
  • The all output wildcard

    • Opening alignments (--open, --append, --new)

    @@ -516,45 +517,145 @@

    -

    Filename substitutions and batch processing (--substitutions, --new, --close, --all)

    +

    Filename substitutions and batch processing (--substitutions, --close, --all)

    -

    --

    +

    + One of the more powerful aspects of Jalview's command line operations is that stores all of the different opened alignment arguments (before opening them) and can apply some arguments to all of the alignments as they are opened. This includes display and output arguments. +

    + +

    + When outputting a file you will obviously want to use a different filename for each of the alignments that are opened. There are several ways you can specify how that filename differs, but the easiest way is to refer to the input filename and change the extension. In the case of an alignment where multiple files are appended, the filename of the first file to be appended or opened is used. +

    + +

    + To refer to different parts of the opening filename, you can use the strings +

    + The braces (curly brackets '{', '}') are important! +

    + +

    + These filename substitutions are on by default, but if for whatever reason you wish to disable the substitutions, they can be turned off (or back on again) through the list of arguments with: +

    + +

    --substitutions / --nosubstitutions

    + Enable (or disable) filename substitutions in the following argument values and sub-value modifier values. 

    +  jalview --open exampes/uniref50.fa --nosubstitutions --output I_Want_A_Weird_Output_Filname_With_{basename}_Curly_Brackets_In_And_A_Sensible_One_Next.stk --substitutions --output tmp/{basename}.stk --headless

    -

    --

    +

    + For opening single files this is less useful, since you could obviously just type the output filename, but for multiple opened alignments you can also use these substituted values and they will be replaced by the relevant part of the filename given for each opened alignment window. Normally an --output or --image argument will only be applied to the latest opened alignment window, but you can tell Jalview to apply some arguments to all alignments that have been opened (so far) by using the --all argument. +

    + +

    --all / -noall

    + +

    + When using the --all argument, following arguments will apply to all of the previously opened alignment windows. You can turn this behaviour off for following arguments using the --noall argument. The arguments that can apply to all previously opened alignments are: +
    + --colour +
    + --sortbytree +
    + --showannotations +
    + --wrap +
    + --nostructure +
    + --notempfac +
    + --showssannotations +
    + --image +
    + --type +
    + --textrenderer +
    + --scale +
    + --width +
    + --height +
    + --output +
    + --format +
    + --groovy +
    + --backups +
    + --overwrite +
    + --close +

    +

    + In particular, for our example above, we can use -all and --output like this (--close will be explained in a moment): + 

    +  jalview --open all_my_fasta_files/*.fa --all --output all_my_converted_stockholm_files/{basename}.stk --close --headless
+
    +

    + Another example would be to create a print quality coloured postscript image for every Fasta file in several directories and place the image next to the alignment: + 

    +  jalview --open */*.fa --all --colour gecos-flower --image {dirname}/{basename}.eps --textrenderer text --close --headless
+
    + or thumbnails for every Fasta file with 
    +  jalview --open */*.fa --all --colour gecos-flower --image {dirname}/{basename}.png --width 256 --height 256 --close --headless

    -

    --

    +

    --close

    + The --close tag is used to close an alignment window after all output files or image exports are performed. This reduces memory use, especially if an --open value is set to open many files. These will be opened, formatted and output sequentially so since they are closed before the next one is opened memory use will not build up over a large number of alignments.

    -

    --

    + +

    The all output wildcard: --output "*.ext", --image "*.ext"

    + +

    + Purely as an intuitive syntactic sweetener, you can use the --output wildcard * at the beginning of the output filename as shorthand for --all --output {dirname}/{basename} followed by whatever you put after the *. For example, to achieve the same as the thumbnails example above, you could use + 

    +  jalview --open */*.fa --image "*.png" --colour gecos-flower --width 256 --height 256 --close --headless
+
    + Here we move the --colour argument after the --output argument (it will still be applied before the image export or file output) so that it is included after the implied --all argument. The thumbnails will be placed in the same directory as the alignment file with the same filename except for a different extension of .png. +

    + For a simple conversion of many fasta files into Stockholm format, simply use 

    +  jalview --open all_my_fasta_files/*.fa --output "*.stk" --close --headless

    -

    --

    +

    + Important! Please note that using a bareword *.ext (i.e. without an escape or quotation marks) in a shell command line will potentially be expanded by the shell to a list of all the files in the current directory ending with .ext before this value is passed to Jalview. This will result in the first of these files being overwritten (multiple times)! If you shell-escape the * (usually with a backslash '\') or enclose the value in quotation marks ('"*.ext"') then the shell will not expand the wildcard. +

    + An alternative is to use an equals sign ('=') with no spaces between the argument and the value, --output=*.ext, which Jalview will interpret the same, but the shell will not automatically expand before it is sent to Jalview, e.g. 

    +  jalview --open all_my_fasta_files/*.fa --output=*.stk --close --headless

    +
    Continue to Jalview Command Line Arguments: advanced usage. - +
    + Return to Jalview Command Line Arguments: summary. diff --git a/help/help/html/features/clarguments-intro.html b/help/help/html/features/clarguments-intro.html index cee0a04..c3cc918 100644 --- a/help/help/html/features/clarguments-intro.html +++ b/help/help/html/features/clarguments-intro.html @@ -78,7 +78,10 @@ running Jalview from the command line.

    +
    Continue to Jalview Command Line Arguments: basic usage. +
    + Return to Jalview Command Line Arguments: summary. diff --git a/help/help/html/features/clarguments.html b/help/help/html/features/clarguments.html index 78fb006..944b1aa 100644 --- a/help/help/html/features/clarguments.html +++ b/help/help/html/features/clarguments.html @@ -98,9 +98,14 @@ + ‑‑threads NUMBER + When opening multiple alignment windows, set a limit of NUMBER alignments being processed at one time. The default is 3. + + + ‑‑jvmmempc=PERCENT - Limit maximum heap size (memory) to PERCENT% of total physical memory detected. + Limit maximum heap size (memory) to PERCENT% of total physical memory detected. This defaults to 90 if total physical memory can be detected.
    The equals sign ("=") separator must be used with no spaces. @@ -112,7 +117,7 @@ ‑‑jvmmemmax=MAXMEMORY - Limit maximum heap size (memory) to MAXMEMORY. MAXMEMORY can be specified in bytes, kilobytes(k), megabytes(m), + Limit maximum heap size (memory) to MAXMEMORY. MAXMEMORY can be specified in bytes, kilobytes(k), megabytes(m), gigabytes(g) or if you're lucky enough, terabytes(t). This defaults to 32g if total physical memory can be detected, or to 8g if total physical memory cannot be detected.
    diff --git a/src/jalview/bin/Commands.java b/src/jalview/bin/Commands.java index 33bb55e..8277d92 100644 --- a/src/jalview/bin/Commands.java +++ b/src/jalview/bin/Commands.java @@ -15,6 +15,8 @@ import java.util.Map; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.RejectedExecutionHandler; import java.util.concurrent.ThreadPoolExecutor; @@ -167,6 +169,7 @@ public class Commands progress = System.currentTimeMillis()); progressBarSet = true; } + List> futures = new ArrayList<>(); for (String id : argParser.getLinkedIds()) { @@ -193,7 +196,7 @@ public class Commands return null; }; - executor.submit(process); + futures.add(executor.submit(process)); Console.debug( "Running " + executor.getActiveCount() + " processes."); } @@ -214,6 +217,20 @@ public class Commands desktop.setProgressBar(null, progress); } + // wait for all to process + for (Future f : futures) + { + try + { + f.get(); + } catch (InterruptedException e1) + { + Console.warn("Processes were interrupted.", e1); + } catch (ExecutionException e2) + { + Console.warn("Processes failed.", e2); + } + } } if (argParser.getBootstrapArgs().getBoolean(Arg.QUIT)) { diff --git a/src/jalview/bin/argparser/Arg.java b/src/jalview/bin/argparser/Arg.java index 73dec4e..384e843 100644 --- a/src/jalview/bin/argparser/Arg.java +++ b/src/jalview/bin/argparser/Arg.java @@ -198,7 +198,7 @@ public enum Arg UNSETARGFILE( "Unsets the current value of the argfilename. Inserted after argfile contents.", Opt.UNARY, Opt.LINKED, Opt.MULTI, Opt.PRIVATE, Opt.NOACTION), - THREADS("When opening multiple alignment windows, set a limit to alignments being processed at one time.", + THREADS("When opening multiple alignment windows, set a limit to alignments being processed at one time. The default is 3.", Opt.BOOTSTRAP, Opt.STRING, Opt.NODUPLICATEVALUES, Opt.NOACTION), // these last two have no purpose in the normal Jalview application but are