(proxy) [jalal@goku apex]$ export PATH=/usr/local/cuda-11./bin:$PATH 
(proxy) [jalal@goku apex]$ export LD_LIBRARY_PATH=/usr/local/cuda-11.1/lib64:$LD_LIBRARY_PATH
(proxy) [jalal@goku apex]$ pip uninstall torchaudio torch torchvision
Found existing installation: torchaudio 0.9.0
Uninstalling torchaudio-0.9.0:
  Would remove:
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchaudio-0.9.0.dist-info/*
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchaudio/*
Proceed (Y/n)? y
  Successfully uninstalled torchaudio-0.9.0
Found existing installation: torch 1.9.0+cu111
Uninstalling torch-1.9.0+cu111:
  Would remove:
    /scratch3/venv/proxy/bin/convert-caffe2-to-onnx
    /scratch3/venv/proxy/bin/convert-onnx-to-caffe2
    /scratch3/venv/proxy/lib/python3.8/site-packages/caffe2/*
    /scratch3/venv/proxy/lib/python3.8/site-packages/torch-1.9.0+cu111.dist-info/*
    /scratch3/venv/proxy/lib/python3.8/site-packages/torch/*
Proceed (Y/n)? y
  Successfully uninstalled torch-1.9.0+cu111
Found existing installation: torchvision 0.10.0+cu111
Uninstalling torchvision-0.10.0+cu111:
  Would remove:
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision-0.10.0+cu111.dist-info/*
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision.libs/libcudart.05b13ab8.so.11.0
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision.libs/libjpeg.ceea7512.so.62
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision.libs/libnvjpeg.d463bc90.so.11
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision.libs/libpng16.7f72a3c5.so.16
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision.libs/libz.1328edc3.so.1
    /scratch3/venv/proxy/lib/python3.8/site-packages/torchvision/*
Proceed (Y/n)? y
  Successfully uninstalled torchvision-0.10.0+cu111
(proxy) [jalal@goku apex]$ pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu111
  Using cached https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp38-cp38-linux_x86_64.whl (2041.3 MB)
Collecting torchvision==0.10.0+cu111
  Using cached https://download.pytorch.org/whl/cu111/torchvision-0.10.0%2Bcu111-cp38-cp38-linux_x86_64.whl (23.2 MB)
Collecting torchaudio==0.9.0
  Using cached torchaudio-0.9.0-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
Requirement already satisfied: typing-extensions in /scratch3/venv/proxy/lib/python3.8/site-packages (from torch==1.9.0+cu111) (3.10.0.2)
Requirement already satisfied: numpy in /scratch3/venv/proxy/lib/python3.8/site-packages (from torchvision==0.10.0+cu111) (1.21.2)
Requirement already satisfied: pillow>=5.3.0 in /scratch3/venv/proxy/lib/python3.8/site-packages (from torchvision==0.10.0+cu111) (8.3.2)
Installing collected packages: torch, torchvision, torchaudio
Successfully installed torch-1.9.0+cu111 torchaudio-0.9.0 torchvision-0.10.0+cu111
(proxy) [jalal@goku apex]$  pip install -v --disable-pip-version-check --no-cache-dir > --global-option="--cpp_ext" --global-option="--cuda_ext" ./
/scratch3/venv/proxy/lib/python3.8/site-packages/pip/_internal/commands/install.py:229: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
    Running command python setup.py egg_info


    torch.__version__  = 1.9.0+cu111


    running egg_info
    creating /scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info
    writing /scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/PKG-INFO
    writing dependency_links to /scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/dependency_links.txt
    writing top-level names to /scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/top_level.txt
    writing manifest file '/scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/SOURCES.txt'
    reading manifest file '/scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/SOURCES.txt'
    writing manifest file '/scratch/tmp/pip-pip-egg-info-balgcp4g/apex.egg-info/SOURCES.txt'
    /scratch/tmp/pip-req-build-rqcjg2ac/setup.py:67: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
      warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
    Running command /scratch3/venv/proxy/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/scratch/tmp/pip-req-build-rqcjg2ac/setup.py'"'"'; __file__='"'"'/scratch/tmp/pip-req-build-rqcjg2ac/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cuda_ext install --record /scratch/tmp/pip-record-jpa0j20t/install-record.txt --single-version-externally-managed --compile --install-headers /scratch3/venv/proxy/include/site/python3.8/apex


    torch.__version__  = 1.9.0+cu111


    /scratch/tmp/pip-req-build-rqcjg2ac/setup.py:67: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
      warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

    Compiling cuda extensions with
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2018 NVIDIA Corporation
    Built on Sat_Aug_25_21:08:01_CDT_2018
    Cuda compilation tools, release 10.0, V10.0.130
    from /usr/local/cuda-10.0/bin

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/scratch/tmp/pip-req-build-rqcjg2ac/setup.py", line 159, in <module>
        check_cuda_torch_binary_vs_bare_metal(CUDA_HOME)
      File "/scratch/tmp/pip-req-build-rqcjg2ac/setup.py", line 99, in check_cuda_torch_binary_vs_bare_metal
        raise RuntimeError("Cuda extensions are being compiled with a version of Cuda that does " +
    RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.  Pytorch binaries were compiled with Cuda 11.1.
    In some cases, a minor-version mismatch will not cause later errors:  https://github.com/NVIDIA/apex/pull/323#discussion_r287021798.  You can try commenting out this check (at your own risk).
ERROR: Command errored out with exit status 1: /scratch3/venv/proxy/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/scratch/tmp/pip-req-build-rqcjg2ac/setup.py'"'"'; __file__='"'"'/scratch/tmp/pip-req-build-rqcjg2ac/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cuda_ext install --record /scratch/tmp/pip-record-jpa0j20t/install-record.txt --single-version-externally-managed --compile --install-headers /scratch3/venv/proxy/include/site/python3.8/apex Check the logs for full command output.
(proxy) [jalal@goku apex]$