Skip to content

LECASofts / LECABashLib

Go to a project
Toggle navigation
Toggle navigation pinning
L

LECABashLib

A Bash library containing function helping the development of bash script mainly for writing jobs for the LUKE cluster

Star
0

The LECA Bash Library

The LECA Bash Library aims to provide a set of function helping the development of bash scripts used to submit jobs on the computation grid.

The LECA Bash Library is constituted by a set of bash scripts that can be sourced in your script or directly from the interactive shell if you want to use the functionalities of the LECA Bash Library from your interactive session.

For activating the LECA Bash Library you must source the util.sh bash script.

    > source $LECABASHHOME/utils.sh

once activated modules can be loaded using the include command.

    > include timeout

The include command is aware about already loaded modules and takes care to not reload several time the same module. To force the reloading of a module, use the reload command.

    > reload timeout

The module is reloaded with all its dependancies. A logging message advertise the name of each reloaded module.

Modules provided by the LECA Bash Library

  • atexit : for automatically execute task at the end of the execution
  • clobber : for managing the clobber/noclobber status
  • download : for managing download from FTP and Web including MD5 checks
  • ifs : for managing the input field separator
  • lecaluke : declare variable related for the LECA computers in LUKE
  • logging : adds functionnalities for managing log files and log levels
  • mutex : manages concurent access to resources through locks and semaphores
  • sets : manages set of values
  • stacks : manages stacks of values
  • tempdir : manages the creation of temporary directories and their erasements
  • timeout : provides a function for limiting the maximum execution time of a command
  • irods : provide functions wrapping the iRods iget and iput icommands.

Detailed description of modules

atexit

Activating the module

    > source $LECABASHHOME/utils.sh
    > include atexit

Functions provided:

registeratexit

Allows to register commands that will be automatically executed at the end of the script execution.

    > include atexit
    > registeratexit "echo this is the end of the script"

clobber

Manages the clobber/noclobber property of the bash shell

When the noclobber mode is on you cannot substitute an existing file by a empty one using a UNIX redirection.

Activating the module

    > source $LECABASHHOME/utils.sh
    > include clobber

Functions provided:

setnoclobber

Sets the noclobber mode on.

You cannot erase an existing file by a redirection

    > include clobber
    > setnoclobber
    > getnoclobber
    on

setclobber

Sets the noclobber mode off.

You can erase an existing file by a redirection

    > include clobber
    > setclobber
    > getnoclobber
    off

getnoclobber

    getnoclobber

Returns the noclobber status as a string

  • on: the noclobber mode is on
  • off: the noclobber mode is off
    > include clobber
    > getnoclobber
    off

pushclobber

    pushclobber

Saves the current noclobber mode on a stack and sets it to 'off'

pushnoclobber

    pushnoclobber

Saves the current noclobber mode on a stack and sets it to 'on'

popclobber

    popclobber

Restores the last pushed noclobber mode

popnoclobber

    popnoclobber

Alias for the popclobber function. Using one of the both commands popnoclobber or popclobber leads to the same action, restoring the last pushed noclober mode

download

Activating the module

    > source $LECABASHHOME/utils.sh
    > include download

Functions provided:

download

    download <URL> [USERNAME] [PASSWORD]

Downloads the file specified by URL. If specifed the second (USERNAME) and third (PASSWORD) parametters allow to indicate a login and password for a protected connection

downloadandcheck

    downloadandcheck <URL> <MD5URL> <DESTINATION> [USERNAME] [PASSWORD]

Downloads the file specified by URL and check file integrity. The MD5 checksum file URL is indicated using the MD5URL parametters. You must specified a destination filename to store the downloaded data using the DESTINATION parametter. The MD5 file is downloaded under the name DESTINATION.md5 If specifed the fourth (USERNAME) and fifth (PASSWORD) parametters allow to indicate a login and password for a protected connection

On error the function has a return status set to 1, 0 otherwise.

setmaxretry

    setmaxretry <MAXRETRY>

If downloading failed the downloadandcheck function can retry to download the file several times before a complete failure. This function allows to define the maximum of time the downloadandcheck tries to redownload before exiting in error.

ifs

Activating the module

    > source $LECABASHHOME/utils.sh
    > include ifs

Functions provided:

pushifs

    pushifs <NEWIFS>

popifs

    popifs

lecaluke

The lecaluke module provides a set of variable defining emplacement where LECA ressources are stored on the LUKE cluster.

Activating the module

    > source $LECABASHHOME/utils.sh
    > include lecaluke

Functions provided:

latesttaxonomy

    latesttaxonomy

Defined variables

LECA_ENVIRONEMENT

LOCK_DIR

DATA_DIR

BIODATA_DIR

TAXONOMY_DIR

TREMBL_DIR

SWISSPROT_DIR

logging

The logging module define a set of function allowing to log on the standard error or into a file messages.

All the log will be done on file descriptor 3 By default file descriptor 3 is redirected to stderr (2) By default the logging level is set to WARNING. Therefore only WARNING and ERROR messages will be actually logged.

Activating the module

    > source $LECABASHHOME/utils.sh
    > include logging

Functions provided:

logdebug

    logdebug <MESSAGE>

Writes MESSAGE as a debug level log to the current log file.

loginfo

    loginfo <MESSAGE>

Writes MESSAGE as a info level log to the current log file.

logerror

    logerror <MESSAGE>

Writes MESSAGE as a error level log to the current log file.

logwarning

    logwarning <MESSAGE>

Writes MESSAGE as a warning level log to the current log file.

setloglevel

    setloglevel <LEVEL>

Set the current logging level to the level specified by LEVEL. Level is DEBUG, INFO, WARNING or ERROR. A log is recorded only when it is submitted with a level greater or equal to the current logging level.

openlogfile

    openlogfile <FILENAME>

Redirected all logging to the file specified by FILENAME. If the file already exists, new logs are appened at the end of the file. If the file does not exist it is created

closelogfile

    closelogfile

Closes the current logfile and redirect the logging to stderr.

mutex

Activating the module

    > source $LECABASHHOME/utils.sh
    > include mutex

Functions provided:

getlock

    getlock <LOCKNAME> [global]

releaselock

    releaselock <LOCKNAME> [global]

getsemaphore

    getsemaphore <SEMAPHORENAME> <MAXCOUNT> [global]

releasesemaphore

    releasesemaphore <SEMAPHORENAME> [global]

releaseoldlock

    releaseoldlock [MINUTES]

Variable provided

LOCK_DIR

LECA_LOCK_DELAY

LECA_LOCK_MAXRETRY

sets

Activating the module

    > source $LECABASHHOME/utils.sh
    > include sets

Functions provided:

newset

    newset <SETNAME>

setsize

    setsize <SETNAME>

setaddvalue

    setsize <SETNAME> <VALUE>

setdelvalue

    setdelvalue <SETNAME> <VALUE>

setcontains

    setcontains <SETNAME> <VALUE>

setunion

    setunion <SETNAME1> <SETNAME2> <DESTINATION>

setintersec

    setintersec <SETNAME1> <SETNAME2> <DESTINATION>

stack

Activating the module

    > source $LECABASHHOME/utils.sh
    > include stack

Functions provided:

newstack

    newstack <STACKNAME>

Creates a new stack named STACKNAME.

stacksize

    stacksize <STACKNAME>

Returns the count of element stored in the stack.

pushvalue

    pushvalue <STACKNAME> <VALUE>

Adds the value VALUE at the end of the stack STACKNAME.

popvalue

    popvalue <STACKNAME> <VARIABLE>

Pops out the last pushed value of the stack and store it into the VARIABLE. If the stack is already empty the function returns nothing and the return status is set to 1 instead of 0 usually.

tempdir

Activating the module

    > source $LECABASHHOME/utils.sh
    > include tempdir

Functions provided:

availabledisk

    availabledisk <DIRPATH>

registertmpdir

    registertmpdir <DIRPATH>

getbasetmpdir

    getbasetmpdir

tempdirectory

    tempdirectory <VARIABLE> [MINSIZE]

timeout

Activating the module

    > source $LECABASHHOME/utils.sh
    > include timeout

Functions provided:

timeoutcmd

    timeoutcmd <SECONDE> <UNIX COMMAND>

irods

The irods module define a set of function wrapping the classical iput and iget commands.

Activating the module

    > source $LECABASHHOME/utils.sh
    > include irods

Functions provided:

isetressource

    isetresource <RESOURCE>

Set the default resource used by iput to store data to RESOURCE.

    iresourceexists <IROD_SOURCE>

Function returning an error status according if the specified iRods resource exists or not.

  • Returns an error status set to 0 if the resource exists,
  • Returns an error status set to 1 otherwise.

ex:

        if iresourceexists $ipath ; then
              echo $ipath exists
        else
              echo $ipath is absent
        fi

igetwithmd5

    igetwithmd5 <IROD_SOURCE> [LOCAL_DEST]

Gets an iRods resource and check if the uploaded file is compatible with the MD5 checksum stored on iRods to. The MD5 file must have the same name than the downloaded resource with the .md5 extension added.

The LOCAL_DEST parameter is optional and without it the downloaded file is saved in the current working directory.

The function download the file until the MD5 check is ok or the maximum retrial count is reached.

igetwithmd5 is not taking care of protecting is access by a lock or a semaphore.