# The *LECA Bash Library* The LECA Bash Library aims to provide a set of function helping the development of bash scripts used to submit jobs on the computation grid. The LECA Bash Library is constituted by a set of bash scripts that can be sourced in your script or directly from the interactive shell if you want to use the functionalities of the LECA Bash Library from your interactive session. For activating the *LECA Bash Library* you must source the `lecabashlib.sh` bash script. ```{shell} > source $LECABASHHOME/lecabashlib.sh ``` once activated modules can be loaded using the `include` command. ```{shell} > include timeout ``` The include command is aware about already loaded modules and takes care to not reload several time the same module. To force the reloading of a module, use the `reload` command. ```{shell} > reload timeout ``` The module is reloaded with all its dependancies. A logging message advertise the name of each reloaded module. You can specify alternative place where to look for non standard modules by setting the `LECABASHLIB_PATH` environment variable. If several paths have to be declared, you must separate them using colon ```{shell} > export LECABASHLIB_PATH="/new/path/1:/new/path/2:${LECABASHLIB_PATH}" ``` ---------------------------------------------------------- # Modules provided by the *LECA Bash Library* - **[atexit](#atexit)** : for automatically execute task at the end of the execution - **[clobber](#clobber)** : for managing the clobber/noclobber status - **[download](#download)** : for managing download from FTP and Web including MD5 checks - **[ifs](#ifs)** : for managing the *input field separator* - **[irods](#irods)** : provide functions wrapping the iRods iget and iput icommands. - **[lecaluke](#lecaluke)** : declare variable related for the LECA computers in LUKE - **[logging](#logging)** : adds functionnalities for managing log files and log levels - **[mutex](#mutex)** : manages concurent access to resources through locks and semaphores - **[sets](#sets)** : manages *set* of values - **[signal](#signal)** : manages signal handler functions - **[stacks](#stacks)** : manages *stacks* of values - **[tempdir](#tempdir)** : manages the creation of temporary directories and their erasements - **[timeout](#timeout)** : provides a function for limiting the maximum execution time of a command - **[xml](#xml)** : provides a set of functions to write basic XML files ---------------------------------------------------------- # Detailed description of modules ## atexit ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include atexit ``` ### Functions provided: #### registeratexit Allows to register commands that will be automatically executed at the end of the script execution. ```{shell} > include atexit > registeratexit "echo this is the end of the script" ``` ---------------------------------------------------------- ## clobber Manages the **clobber**/**noclobber** property of the bash shell When the **noclobber** mode is `on` you cannot substitute an existing file by a empty one using a UNIX redirection. ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include clobber ``` ### Functions provided: #### setnoclobber Sets the **noclobber** mode on. You cannot erase an existing file by a redirection ```{shell} > include clobber > setnoclobber > getnoclobber on ``` #### setclobber Sets the **noclobber** mode off. You can erase an existing file by a redirection ```{shell} > include clobber > setclobber > getnoclobber off ``` #### getnoclobber ```{shell} getnoclobber ``` Returns the noclobber status as a string - `on`: the noclobber mode is on - `off`: the noclobber mode is off ```{shell} > include clobber > getnoclobber off ``` #### pushclobber ```{shell} pushclobber ``` Saves the current noclobber mode on a stack and sets it to 'off' #### pushnoclobber ```{shell} pushnoclobber ``` Saves the current noclobber mode on a stack and sets it to 'on' #### popclobber ```{shell} popclobber ``` Restores the last pushed noclobber mode #### popnoclobber ```{shell} popnoclobber ``` Alias for the `popclobber` function. Using one of the both commands `popnoclobber` or `popclobber` leads to the same action, restoring the last pushed noclober mode ---------------------------------------------------------- ## download ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include download ``` ### Functions provided: #### download ```{shell} download [USERNAME] [PASSWORD] ``` Downloads the file specified by `URL`. If specifed the second (`USERNAME`) and third (`PASSWORD`) parametters allow to indicate a login and password for a protected connection #### downloadandcheck ```{shell} downloadandcheck [USERNAME] [PASSWORD] ``` Downloads the file specified by `URL` and check file integrity. The MD5 checksum file URL is indicated using the `MD5URL` parametters. You must specified a destination filename to store the downloaded data using the `DESTINATION` parametter. The MD5 file is downloaded under the name DESTINATION.md5 If specifed the fourth (`USERNAME`) and fifth (`PASSWORD`) parametters allow to indicate a login and password for a protected connection On error the function has a return status set to *1*, *0* otherwise. #### setmaxretry ```{shell} setmaxretry ``` If downloading failed the downloadandcheck function can retry to download the file several times before a complete failure. This function allows to define the maximum of time the downloadandcheck tries to redownload before exiting in error. ---------------------------------------------------------- ## ifs ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include ifs ``` ### Functions provided: #### pushifs ```{shell} pushifs ``` #### popifs ```{shell} popifs ``` ---------------------------------------------------------- ## lecaluke The lecaluke module provides a set of variable defining emplacement where LECA ressources are stored on the LUKE cluster. ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include lecaluke ``` ### Functions provided: #### latesttaxonomy ```{shell} latesttaxonomy ``` ### Defined variables #### LECA_ENVIRONEMENT #### LOCK_DIR #### DATA_DIR #### BIODATA_DIR #### TAXONOMY_DIR #### TREMBL_DIR #### SWISSPROT_DIR ---------------------------------------------------------- ## logging The *logging* module define a set of function allowing to log on the standard error or into a file messages. All the log will be done on file descriptor 3 By default file descriptor 3 is redirected to stderr (2) By default the logging level is set to `WARNING`. Therefore only `WARNING` and `ERROR` messages will be actually logged. ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include logging ``` ### Functions provided: #### logdebug ```{shell} logdebug ``` Writes `MESSAGE` as a debug level log to the current log file. #### loginfo ```{shell} loginfo ``` Writes `MESSAGE` as a info level log to the current log file. #### logerror ```{shell} logerror ``` Writes `MESSAGE` as a error level log to the current log file. #### logwarning ```{shell} logwarning ``` Writes `MESSAGE` as a warning level log to the current log file. #### setloglevel ```{shell} setloglevel ``` Set the current logging level to the level specified by `LEVEL`. Level is `DEBUG`, `INFO`, `WARNING` or `ERROR`. A log is recorded only when it is submitted with a level greater or equal to the current logging level. #### openlogfile ```{shell} openlogfile ``` Redirected all logging to the file specified by `FILENAME`. If the file already exists, new logs are appened at the end of the file. If the file does not exist it is created #### closelogfile ```{shell} closelogfile ``` Closes the current logfile and redirect the logging to stderr. ---------------------------------------------------------- ## mutex ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include mutex ``` ### Functions provided: #### getlock ```{shell} getlock [global] ``` #### releaselock ```{shell} releaselock [global] ``` #### getsemaphore ```{shell} getsemaphore [global] ``` #### releasesemaphore ```{shell} releasesemaphore [global] ``` #### releaseoldlock ```{shell} releaseoldlock [MINUTES] ``` ### Variable provided #### LOCK_DIR #### LECA_LOCK_DELAY #### LECA_LOCK_MAXRETRY ---------------------------------------------------------- ## sets ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include sets ``` ### Functions provided: #### newset ```{shell} newset ``` #### setsize ```{shell} setsize ``` #### setaddvalue ```{shell} setsize ``` #### setdelvalue ```{shell} setdelvalue ``` #### setcontains ```{shell} setcontains ``` #### setunion ```{shell} setunion ``` #### setintersec ```{shell} setintersec ``` ---------------------------------------------------------- ## signal ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include signal ``` ### Functions provided: #### registertrap ```{shell} registertrap ``` #### run ```{shell} run ``` ---------------------------------------------------------- ## stacks ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include stacks ``` ### Functions provided: #### newstack ```{shell} newstack ``` Creates a new stack named `STACKNAME`. #### stacksize ```{shell} stacksize ``` Returns the count of element stored in the stack. #### pushvalue ```{shell} pushvalue ``` Adds the value VALUE at the end of the stack `STACKNAME`. #### popvalue ```{shell} popvalue ``` Pops out the last pushed value of the stack and store it into the `VARIABLE`. If the stack is already empty the function returns nothing and the return status is set to *1* instead of *0* usually. ---------------------------------------------------------- ## tempdir ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include tempdir ``` ### Functions provided: #### availabledisk ```{shell} availabledisk ``` #### registertmpdir ```{shell} registertmpdir ``` #### getbasetmpdir ```{shell} getbasetmpdir ``` #### tempdirectory ```{shell} tempdirectory [MINSIZE] ``` ---------------------------------------------------------- ## timeout ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include timeout ``` ### Functions provided: #### timeoutcmd ```{shell} timeoutcmd ``` ---------------------------------------------------------- ## path ### Activating the module > source $LECABASHHOME/lecabashlib.sh > include path ### Functions provided: #### relative2absolute relative2absolute #### absolute2relative ```bash absolute2relative ``` ---------------------------------------------------------- ## irods The *irods* module define a set of function wrapping the classical `iput` and `iget` commands. ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include irods ``` ### Functions provided: #### isetressource ```{shell} isetresource ``` Set the default resource used by `iput` to store data to `RESOURCE`. ---------------------------------------------------------- ```{shell} iresourceexists ``` Function returning an error status according if the specified iRods resource exists or not. - Returns an error status set to 0 if the resource exists, - Returns an error status set to 1 otherwise. ex: ```{shell} if iresourceexists $ipath ; then echo $ipath exists else echo $ipath is absent fi ``` ---------------------------------------------------------- #### igetwithmd5 ```{shell} igetwithmd5 [LOCAL_DEST] ``` Gets an iRods resource and check if the uploaded file is compatible with the MD5 checksum stored on iRods to. The MD5 file must have the same name than the downloaded resource with the `.md5` extension added. The `LOCAL_DEST` parameter is optional and without it the downloaded file is saved in the current working directory. The function download the file until the MD5 check is ok or the maximum retrial count is reached. `igetwithmd5` is not taking care of protecting is access by a lock or a semaphore. ## xml ### Activating the module ```{shell} > source $LECABASHHOME/lecabashlib.sh > include xml ``` ### Functions provided: #### XML_header ```{shell} XML_header ``` #### XML_push ```{shell} XML_push ``` #### XML_pop ```{shell} XML_pop ``` #### XML_tag_data ```{shell} XML_tag_data ``` #### XML_tag_attr_data ```{shell} XML_tag_attr_data -- ``` #### XML_empty_tag ```{shell} XML_empty_tag ``` #### XML_data ```{shell} XML_data ```