runWiscJobs: a tool for using HTCondor and HDFS

To use runWiscJobs, first obtain an OSG grid certificate. If you are a member of a large collaboration like CMS, LSST, or LZ, you should register with a Virtual Organization (VO), but if you are a member of a small experiment without a VO, you can specify the GLOW VO when you need to provide one. Follow the instructions under How to get a Grid User Certificate here:

https://wiki.physics.wisc.edu/lz/index.php/ComputingDocumentation

Log into a login machine:

ssh login.hep.wisc.edu

Obtain a proxy:

voms-proxy-init -valid 142:00

or if you have a VO:

voms-proxy-init -valid 142:00  -voms VO_NAME

where VO_NAME is the name of your VO.

Then you can use runWiscJobs.py to run your jobs with HTCondor:

 python /usr/local/bin/runWiscJobs.py \
        --WorkFlow="test.mac_test1" \
        --Executable="/afs/hep.wisc.edu/user/cvuosalo/work/lz/RunLUXSim.sh" \
        --Experiment=lz \
        --nJobs=10 \
        --TransferInputFiles="/afs/hep.wisc.edu/user/cvuosalo/work/lz/test.mac" \
	--HDFSProdDir=outfiles \
        --MakeUniqueOutput \
        --UserEnv="UserName=`whoami` " \
        --OutputDir="test.mac" \
        --Arguments="test.mac"

You can see all the arguments for runWiscJobs.py by doing:

 python /usr/local/bin/runWiscJobs.py --help

Note that you need to pass an Experiment name to runWiscJobs.py. It might be lz, lux, miniclean, ricochet, etc.

Executable is your script or program that runs your simulation program.

HDFSProdDir is the directory where your Executable should place the output files you would like copied to HDFS. This directory will be under the working directory on the worker node. If your executable handles on its own the copying of output files to HDFS, set this parameter to “None”.

If your executable generates output files all with the same name, you can use the –MakeUniqueOutput option so that a unique tag gets added to each output file name when it is copied into HDFS.

If your jobs are successful, the output appears under to /hdfs//user//.

Diagnostic logs appear under /nfs_scratch//.