so, $ sbatch torch_gpu_sanity_venv385-11.slurm runs the stuff inside the .slurm except running the /research/jalal/slurm/torch_gpu_sanity_venv385-11.bash line
[jalal@goku fashion_compatibility]$ cat slurm-28334.out 
Mon Sep 13 20:06:44 EDT 2021 ivcgpu10.bu.edu env
CUDA_HOME=/usr/local/cuda-10.0
BASEPATH=/home/grad3/jalal/.local/bin:/opt/rh/rh-php70/root/bin:/scratch2/system/opt/pycharm-community-2018.1.2/bin:/scratch2/google-cloud-sdk/bin:/scratch2/google-cloud-sdk/bin:/scratch3/.cargo/bin:/home/grad3/jalal/.gdrive-downloader:/usr/local/cuda-10.0/bin:/scratch3/MATLAB/matlab/bin:/home/grad3/jalal/.local/bin:/opt/rh/rh-php70/root/bin:/scratch2/system/opt/pycharm-community-2018.1.2/bin:/scratch2/google-cloud-sdk/bin:/scratch2/google-cloud-sdk/bin:/usr/lib64/qt-3.3/bin:/scratch/sjn-p3/anaconda/anaconda3/condabin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/IT/bin:/usr/bin/python3:/home/grad3/jalal/bin:/scratch/sjn-p2/anaconda/anaconda2/bin
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=7
PATH=/home/grad3/jalal/.gdrive-downloader:/usr/local/cuda-10.0/bin:/scratch3/MATLAB/matlab/bin:/home/grad3/jalal/.local/bin:/opt/rh/rh-php70/root/bin:/scratch2/system/opt/pycharm-community-2018.1.2/bin:/scratch2/google-cloud-sdk/bin:/scratch2/google-cloud-sdk/bin:/scratch3/.cargo/bin:/home/grad3/jalal/.gdrive-downloader:/usr/local/cuda-10.0/bin:/scratch3/MATLAB/matlab/bin:/home/grad3/jalal/.local/bin:/opt/rh/rh-php70/root/bin:/scratch2/system/opt/pycharm-community-2018.1.2/bin:/scratch2/google-cloud-sdk/bin:/scratch2/google-cloud-sdk/bin:/usr/lib64/qt-3.3/bin:/scratch/sjn-p3/anaconda/anaconda3/condabin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/IT/bin:/usr/bin/python3:/home/grad3/jalal/bin:/scratch/sjn-p2/anaconda/anaconda2/bin:/usr/bin/python3
torch vers =      1.8.1+cu111
tensor([[0.6323, 0.2388, 0.8772],
        [0.6798, 0.7087, 0.4152],
        [0.5309, 0.6991, 0.4273],
        [0.6753, 0.0531, 0.5897],
        [0.8470, 0.7392, 0.3246]])
cuda avail =      True
cuda current dev= 0
dev 0 = NVIDIA RTX A6000
and
[jalal@goku fashion_compatibility]$ cat torch_gpu_sanity_venv385-11.slurm
#!/bin/bash
#SBATCH --partition=gpu-L --gres=gpu:1 
# -------------------------> ask for 1 GPU
d=$(date)
h=$(hostname)
echo $d $h env         # show CUDA related Env vars 
env|grep -i cuda
# nvidia-smi
#          actual work
/research/jalal/slurm/torch_gpu_sanity_venv385-11.bash
and
[jalal@goku fashion_compatibility]$ cat torch_gpu_sanity_venv385-11.bash 
#!/bin/bash
source /research/jalal/slurm/venv/fash/bin/activate
python main.py --name both_attn_1e_SA --learned --l2_embed --datadir ../../data/ --all_attend_fc  --use_attend --epochs 1
deactivate 
does anyone know why the 10th line of the .slurm script is not being run?