Usage Guide
Learn how to connect, set up your environment, and run jobs on the Petnica HPC cluster.
Table of Contents
0. Requesting an Account
Before you can connect to the cluster, you need an account. Accounts are not requested by users directly — the request must come from the head of the department (seminar leader) of the person who needs access. If you are a seminar participant or associate who needs an account, talk to your head of department and ask them to submit the request on your behalf.
The head of department sends the request by email to hpclab@petnica.rs, using the format below. One block per person who needs an account.
Name:
Surname:
Programme/seminar:
Student/associate (what is the status of user requesting):
email:
The request must be sent by the head of department, not the prospective user. Once the account is created, you will receive your credentials and can proceed with connecting to the cluster (section 1).
Account requests not sent by a head of department will be ignored. Make sure your department head submits the request on your behalf.
1. Connecting to the Cluster
To access the cluster, you need an SSH connection to the head node. The head node is your entry point — from there, you can submit jobs, manage environments, and interact with the system. Your SSH session runs on the head node; compute jobs run on dedicated compute nodes (c1–c5) via SLURM.
ssh username@hpclab.petnica.rs
Replace username with your cluster account username. You will be prompted for your password or SSH key.
After logging in, verify you're on the head node and check your access to the shared filesystem.
hostname
ls /home
2. Copying Files to the Cluster
Before you can run anything, you need your source code and datasets on the cluster. Your home directory lives on shared network storage, so files you copy to the head node are visible on every compute node. The recommended tool is rsync: unlike a plain copy, it transfers only the parts that changed, can resume an interrupted transfer, and preserves file timestamps and permissions — which matters for large datasets that take a while to upload.
All the rsync commands below are run on your own computer, not on the cluster. rsync opens its own SSH connection to the head node, so you don't need to log in first.
To upload a project directory, point rsync at your local folder and a destination path in your cluster home directory. The trailing slash on the source (./myproject/) means "copy the contents of this folder" into the destination.
rsync -avz --exclude '.git' --exclude '__pycache__' \
./myproject/ username@hpclab.petnica.rs:~/myproject/
-a archive mode (recursive, preserves timestamps and permissions). -v verbose (lists transferred files). -z compresses data in transit to save bandwidth. --exclude skips files you don't want to upload — here the .git history and Python caches.
Datasets are often large, so add the -P flag. It shows a progress bar and keeps partially transferred files, meaning that if the connection drops you can re-run the exact same command and rsync resumes where it left off instead of starting over.
rsync -avzP ./dataset/ username@hpclab.petnica.rs:~/data/dataset/
If a large upload is interrupted, simply run the same command again. rsync compares both sides and only sends what's missing, so re-running is cheap and safe.
After a job finishes, copy output files back by swapping the source and destination — the remote path comes first, your local path second.
rsync -avzP username@hpclab.petnica.rs:~/myproject/results/ ./results/
For a one-off file, scp is simpler: scp report.pdf username@hpclab.petnica.rs:~/. Use rsync whenever you're copying directories or anything large.
3. Loading Miniconda with Module System
The cluster uses an environment modules system to manage software. This allows multiple versions of tools to coexist without conflicts. Miniconda (a lightweight Python distribution) is installed centrally and loaded as a module. When you load it, your PATH is updated to point to Miniconda's binary directory, and you gain access to conda commands.
The Miniconda module is available system-wide. Loading it sets up the conda environment on your shell.
module load miniconda
Add this command to your ~/.bashrc or ~/.zshrc (depending on your shell) so it runs automatically when you log in. Then you won't need to load it manually each time.
Check that conda is available and inspect the current environment.
conda --version
conda info
This will display the conda version and installation details. If you see version information, Miniconda is ready to use.
4. Configuring Conda
Conda's shared installation on the cluster comes with default settings that reference Anaconda's official package repositories. To avoid licensing prompts and use the open-source conda-forge repository instead, you should configure conda for your user account. This is a one-time setup.
If you want to use Anaconda's default channels, you need to accept their Terms of Service once. This writes a setting to your ~/.condarc file and prevents the prompt from appearing each time you create an environment.
conda config --set plugins.auto_accept_tos yes
conda-forge is the community-driven, open-source package repository and avoids any licensing concerns. Set it as your default channel with strict priority so it's checked first. This is the recommended approach for the cluster.
conda config --prepend channels conda-forge
conda config --set channel_priority strict
These commands add conda-forge as your primary package source and set strict priority, meaning conda will search conda-forge first and won't mix packages from different channels (which can cause incompatibilities).
Check your conda configuration to confirm the channels are set correctly.
conda config --show-sources
You should see conda-forge listed first in your user configuration (~/.condarc).
5. Creating a Conda Virtual Environment
A conda environment is an isolated Python installation with its own packages and dependencies. Different projects often need different package versions, and environments let you keep these separate without conflicts. You'll create one environment per project.
Create an environment with a descriptive name. In this example, we'll create an environment called myproject. You can choose any name you prefer.
conda create -n myproject --override-channels -c conda-forge python=3.14
-n myproject names the environment. --override-channels -c conda-forge uses only conda-forge as the package source (avoiding the Anaconda defaults channel entirely). python=3.14 specifies Python version 3.14 (replace with your desired version).
python=3.14 means "any 3.14.x patch release" (fuzzy match). Use python==3.14.0 for exact version matching, or python>=3.14,<3.15 for a range.
Once created, activate your environment to use it. This modifies your shell's PATH so that the environment's Python and packages take precedence.
conda activate myproject
Your prompt should now show (myproject) at the beginning, indicating the environment is active.
View all environments you've created on the cluster.
conda env list
The active environment will be marked with an asterisk (*).
6. Installing Python and pip
When you create a conda environment with python=3.14, Python is automatically installed. pip is Python's package manager and comes bundled with Python. Both are already present in your environment. You use pip to install Python packages from PyPI (Python Package Index).
First, ensure your environment is activated, then check the versions.
python --version
pip --version
You should see Python 3.14.x and a corresponding pip version. If not, activate your environment again.
Keeping pip up-to-date ensures you get the latest features and bug fixes when installing packages.
pip install --upgrade pip
You can install any Python package available on PyPI using pip. For example, to install numpy and scipy:
pip install numpy scipy
Use conda install for packages available in conda-forge (often better tested for scientific computing). Use pip install for packages not in conda or for the latest PyPI releases. Never mix them heavily in the same environment without understanding the implications.
7. Example: Installing PyTorch
PyTorch is a popular deep learning library that can run on CPUs or GPUs. The cluster offers 8 NVIDIA GPUs: GTX 960 on c1–c3, GTX 1060 6GB on c4 and GTX 970 on c5 — two cards per node, except c1 and c5, which have one each while a faulty card is out of service. To use GPU acceleration, you must install the GPU-enabled PyTorch build. PyTorch bundles its own CUDA runtime, so you don't need to install CUDA separately — just the right PyTorch wheel for your GPU's architecture.
The GTX 960 and GTX 970 have compute capability 5.2 (sm_52); the GTX 1060 has 6.1 (sm_61). The Maxwell cards are the limit: they need a PyTorch wheel compiled with sm_50 support (sm_50 kernels run on sm_52). What matters is the CUDA wheel variant, not just the version: cu118, cu121, and cu126 wheels keep sm_50, while newer cu128/cu129 wheels drop it. Install the latest PyTorch from the cu126 wheel index (shown below) — it's the newest variant that still ships sm_50, and it covers sm_61 too, so the same environment runs on any node. If you pick a different wheel, confirm sm_50 is present with torch.cuda.get_arch_list(), or restrict your job to Pascal with --constraint=gtx10.
Create a new environment specifically for PyTorch with Python and pip pre-installed.
conda create -n pytorch-env --override-channels -c conda-forge python=3.14 pip
Activate the environment, then install the latest PyTorch with GPU support from the CUDA 12.6 (cu126) wheel index, which still ships Maxwell kernels.
conda activate pytorch-env
pip install torch --index-url https://download.pytorch.org/whl/cu126
The latest PyTorch release with CUDA 12.6 support. The cu126 index URL is the newest wheel variant that still includes Maxwell (sm_50) kernels, so it works on the GTX 960 and GTX 970 as well as the newer GTX 1060.
Save this as test_gpu.py and submit it as a SBATCH job (see section 8) to verify GPU support on a compute node.
import torch
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
print(f"GPU device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'None'}")
# Simple tensor operation on GPU
if torch.cuda.is_available():
x = torch.randn(1000, 1000).cuda()
y = x @ x
print(f"Matrix multiplication on GPU successful, result shape: {y.shape}")
else:
print("GPU not available, skipping GPU test")
8. Running Jobs with SBATCH
The cluster uses SLURM (Simple Linux Utility for Resource ManageMent) to schedule and manage jobs. An sbatch script is a shell script with SLURM directives (special comments starting with #SBATCH) that specify your job's resource requirements. When you submit the script, SLURM queues the job and runs it on an available compute node.
| Item | Value |
|---|---|
| Compute nodes | c1, c2, c3, c4, c5 (5 nodes total) |
| Partition (queue) | nordeus (default) |
| CPUs per node | 12 logical CPUs (hyperthreaded); 20 on c4 |
| Memory per node | 63000M on every node |
| GPUs per node | GTX 960 on c1–c3, GTX 1060 6GB on c4, GTX 970 on c5 — 2 per node, except c1 and c5, which currently have 1 (a faulty card on each is out of service). 8 GPUs in total. |
| GPU selection | --gres=gpu:1 (any), or by model:
gpu:gtx_960:1, gpu:gtx_970:1,
gpu:gtx_1060:1; by generation:
--constraint=gtx9 / gtx10 |
| Max job time | partition cap 14 days, default 3 days if --time is omitted; per-user ceiling is your QoS — normal 3 days, extended 7 days, long 14 days |
Here's a simple sbatch script template. Replace the placeholders with your actual requirements. This example requests 4 CPUs, 8GB memory, 30 minutes, and runs a Python script.
#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --partition=nordeus
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=00:30:00
#SBATCH --output=%x-%N-%j.out
#SBATCH --error=%x-%N-%j.err
set -euo pipefail
# Load modules and activate environment
source /etc/profile.d/modules.sh 2>/dev/null || true
module load miniconda
conda activate myproject
# Run your Python script
python "${SLURM_SUBMIT_DIR}/my_script.py"
--job-name: name shown in job queue--partition: queue to submit to (nordeus is the default)--nodes/--ntasks: usually both 1 for single-node jobs--cpus-per-task: CPU cores needed (max 12 per node, 20 on c4)--mem: total RAM needed for the job--time: max job duration in HH:MM:SS--output/--error: where to write stdout/stderr
To use the GPU, add the --gres (Generic Resource) directive. This is mandatory — without it, your job won't have access to the GPU even though it's physically present on the node.
#!/bin/bash
#SBATCH --job-name=gpu-job
#SBATCH --partition=nordeus
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --gres=gpu:1
#SBATCH --time=00:30:00
#SBATCH --output=%x-%N-%j.out
set -euo pipefail
source /etc/profile.d/modules.sh 2>/dev/null || true
module load miniconda
conda activate pytorch-env
# Verify GPU access
echo "CUDA_VISIBLE_DEVICES: $CUDA_VISIBLE_DEVICES"
nvidia-smi -L
# Run GPU code
python "${SLURM_SUBMIT_DIR}/my_gpu_script.py"
The --gres=gpu:1 directive is crucial. Without it, cgroups (a Linux containerization feature) will hide /dev/nvidia* devices from your job, and CUDA will appear unavailable even though the GPU exists.
Submit your job to the queue and track its progress.
# Submit job
sbatch job.sbatch
# Check job status (your own jobs)
squeue -u $USER
# View full job details
scontrol show job
# Cancel a job
scancel
When you submit, SLURM prints the job ID (e.g., Submitted batch job 42). Use this ID to check status or cancel the job.
After the job completes, view the output and error logs (if separate).
# View output file (e.g., myjob-c1-42.out)
cat myjob-c1-42.out
# Or tail it to see the last lines
tail myjob-c1-42.out
Keep these in mind when writing sbatch scripts:
- Use
$SLURM_SUBMIT_DIR— SLURM copies your script to the compute node's spool directory, so you can't use relative paths. Always reference input files relative to$SLURM_SUBMIT_DIR(the directory you ransbatchfrom). - Load modules in the script — don't assume your shell's environment carries over. Always include
module load minicondaand activate your conda environment. - Use conda run for batch shells — in non-interactive batch scripts,
conda activatecan silently fall back to the base environment. Useconda run --no-capture-output -n myproject python script.pyinstead for reliability. - Set error handling — start with
set -euo pipefailso the script exits immediately if any command fails (useful for debugging).
Complete Workflow Example: CPU Job
Here's a complete example of creating a job, submitting it, and checking results. Assume you have a Python script analysis.py in your home directory.
#!/bin/bash
#SBATCH --job-name=analysis
#SBATCH --partition=nordeus
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --output=%x-%N-%j.out
set -euo pipefail
source /etc/profile.d/modules.sh 2>/dev/null || true
module load miniconda
conda activate myproject
python "${SLURM_SUBMIT_DIR}/analysis.py"
sbatch analysis.sbatch
# Prints: Submitted batch job 101
squeue -u $USER
# Shows job 101 in the queue
# After it completes, view results
cat analysis-c1-101.out
Complete Workflow Example: GPU Job with PyTorch
Here's a complete example using PyTorch on the GPU. Assume you have train.py in your home directory.
#!/bin/bash
#SBATCH --job-name=pytorch-train
#SBATCH --partition=nordeus
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --gres=gpu:1
#SBATCH --time=02:00:00
#SBATCH --output=%x-%N-%j.out
set -euo pipefail
source /etc/profile.d/modules.sh 2>/dev/null || true
module load miniconda
conda activate pytorch-env
# Verify GPU allocation
echo "GPU Allocation:"
echo "CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES"
nvidia-smi
# Run training script
python "${SLURM_SUBMIT_DIR}/train.py"
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"Compute Capability: {torch.cuda.get_device_capability(0)}")
# Dummy training loop
x = torch.randn(100, 10).to(device)
y = torch.randn(100, 1).to(device)
model = torch.nn.Linear(10, 1).to(device)
for epoch in range(10):
output = model(x)
loss = torch.nn.functional.mse_loss(output, y)
print(f"Epoch {epoch + 1}/10, Loss: {loss.item():.4f}")
print("Training complete!")
sbatch train_gpu.sbatch
# Prints: Submitted batch job 102
squeue -u $USER
# Shows job 102 running on one of the compute nodes
# Check output while running
tail -f pytorch-train-c1-102.out
# After completion, view full results
cat pytorch-train-c1-102.out
9. Troubleshooting
Symptom: torch.cuda.is_available() returns False inside your job, even though you requested a GPU with --gres=gpu:1.
Solution: This usually means SLURM allocated the GPU but cgroups aren't exposing it. Try these steps:
1. Verify the GPU was allocated:
echo "CUDA_VISIBLE_DEVICES: $CUDA_VISIBLE_DEVICES"
nvidia-smi
If CUDA_VISIBLE_DEVICES is empty or nvidia-smi fails, the GPU wasn't allocated.
2. Double-check your sbatch directive:
#SBATCH --gres=gpu:1
Without this, SLURM won't allocate a GPU and cgroups will hide the device.
3. Check PyTorch installation:
python -c "import torch; print(torch.cuda.get_arch_list())"
The output must include sm_50 for the Maxwell GPUs (GTX 960/970); sm_61 covers the GTX 1060 on c4. If the list is empty or missing sm_50, reinstall PyTorch with a compatible wheel.
Symptom: Job fails with "no such environment" or wrong Python version.
Solution: Ensure you're loading the module and using conda run or conda activate correctly in the sbatch script.
source /etc/profile.d/modules.sh 2>/dev/null || true
module load miniconda
conda activate myproject
# Or use conda run (more reliable in batch mode)
conda run --no-capture-output -n myproject python script.py
Symptom: Your job dies partway through and sacct shows it in state OUT_OF_MEMORY (or the log mentions an OOM kill) — even though the node still had free RAM.
Why: The cluster enforces the memory you ask for. Each job is confined to its requested --mem, and a job that tries to use more than that is killed automatically. Asking for a bigger node does not help — only your --mem request matters.
Step 1 — see how much it actually used. After the job ends, check its peak memory (MaxRSS) against what it requested (ReqMem):
seff <jobid>
sacct -j <jobid> --format=JobID,State,ReqMem,MaxRSS,Elapsed
Step 2 — request a bit more than the observed MaxRSS in your sbatch script and resubmit. Always set --mem explicitly so your limit (and your usage stats) are predictable. Remember the per-node limits:
- c1–c5: 63000M max
#SBATCH --mem=24G
Symptom: Job is killed because --time limit was exceeded.
Solution: Increase the --time directive. The nordeus partition caps jobs at 14 days and defaults to 3 days if you omit --time; your own ceiling is your QoS (normal 3 days, extended 7 days, long 14 days). Jobs asking for more than the QoS allows are rejected at submission (DenyOnLimit).
#SBATCH --time=04:00:00
Symptom: CUDA error: no kernel image is available for execution on the device
Solution: The PyTorch wheel you installed doesn't include Maxwell (sm_50) kernels — typically because it's a newer cu128/cu129 wheel that dropped them. Reinstall from the cu126 wheel index, the newest variant that still ships sm_50.
pip install torch --index-url https://download.pytorch.org/whl/cu126
See section 7 for details on GPU compute capability.
For additional issues or questions:
- Check job logs:
cat myjob-nodename-jobid.outandcat myjob-nodename-jobid.err - View full job info:
scontrol show job <jobid> - Consult the official SLURM documentation for scheduler questions