Using iCommands for large datasets

There are some advanced options available when using Yoda. These are especially useful when:

  • you have to download, upload, or manage large data sets (up to GB and more or hundreds to thousands of files);

  • you are experienced in command-line tools.

Using iCommands for large datasets

If you work with large amounts of data, you can use iRODS iCommands. With these tools, you can upload and download large datasets efficiently. They are significantly faster than the Yoda network drive, although learning to use them takes time. Therefore, it is suggested that you only use iCommands if you are comfortable using command-line tools. On Windows or macOS you should also be comfortable setting up and using virtual machines or Docker containers, although Windows WSL2 makes it somewhat easier.

Yoda is based on iRODS technology. iCommands allows you to interact with the iRODS backend of Yoda directly. Therefore, it is possible to transfer data to and from Yoda using the iRODS communication protocol.

You will need to change the example user name to your Yoda user name. Your user name should contain only lowercase characters.

Version of iCommands matters

Your local iCommands version should match the iRODS version on the Yoda server. This is currently version 4.2.12.

Step 1. Installing iCommands

You first need to install client software that supports the iRODS protocol on your PC. Native iRODS iCommands packages are only available on Linux, both for CentOS and Ubuntu.

Windows 10 or Windows 11 users can run the iCommands in the Windows Subsystem for Linux (WSL). We recommend using WSL version 2 (WSL2), since it has fixes for a number of issues in WSL version 1 (WSL1). Installation instructions are here, but in a nutshell (including the distribution that currently works best with iRODS iCommands 4.2.12):

  • start PowerShell as administrator
  • wsl --install -d Ubuntu-18.04

macOS users can run the commands inside a Linux VM or Docker container.

Installing iCommands on CentOS

iRODS iCommands 4.2.12 supports CentOS 7.

Use these commands to install the iCommands package on CentOS 7 and tune the network settings:

sudo yum -y install wget epel-release yum-plugin-versionlock
sudo rpm --import https://packages.irods.org/irods-signing-key.asc
wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
sudo yum -y install irods-runtime-4.2.12 irods-icommands-4.2.12

# Prevent automatic upgrades of these two packages
sudo yum versionlock irods-runtime irods-icommands

# Network tuning, ad hoc and permanent
echo "120"  | sudo tee /proc/sys/net/ipv4/tcp_keepalive_time
echo "30" | sudo tee /proc/sys/net/ipv4/tcp_keepalive_intvl
echo "net.ipv4.tcp_keepalive_time=120" | sudo tee -a /etc/sysctl.d/99-sysctl.conf
echo "net.ipv4.tcp_keepalive_intvl=30" | sudo tee -a /etc/sysctl.d/99-sysctl.conf

Upgrading from an earlier version
If you have put packages irods-runtime and irods-icommands on hold, you should first release the hold. Otherwise the instruction is the same as above.

# Show applied locks, if any
yum versionlock list
yum versionlock delete irods-runtime irods-icommands 

Installing iCommands on Ubuntu
iRODS version 4.2.12 currently supports Ubuntu 18.04 LTS.

Use these commands to install the iCommands package on Ubuntu 18.04 LTS (“bionic”) and tune the network settings.

sudo apt-get update
sudo apt-get install -y --only-upgrade ca-certificates
wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add -
echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/renci-irods.list
sudo apt-get update
sudo apt -y install irods-runtime=4.2.12-1~bionic irods-icommands=4.2.12-1~bionic

# Prevent automatic upgrades of these two packages
sudo apt-mark hold irods-runtime irods-icommands

# Network tuning, ad hoc and permanent
echo "120" | sudo tee /proc/sys/net/ipv4/tcp_keepalive_time
echo "30" | sudo tee /proc/sys/net/ipv4/tcp_keepalive_intvl
echo "net.ipv4.tcp_keepalive_time=120" | sudo tee -a /etc/sysctl.d/99-sysctl.conf
echo "net.ipv4.tcp_keepalive_intvl=30" | sudo tee -a /etc/sysctl.d/99-sysctl.conf

Upgrading from an earlier version
If you have put packages irods-runtime and irods-icommands on hold, you should first release the hold. Otherwise the instruction is the same as above.

# Show applied holds, if any
apt-mark showhold
apt-mark unhold irods-runtime irods-icommands

Step 2. Configuring iCommands

You need to configure the iCommands to connect to the right Yoda environment. When running in a virtual machine or docker container you probably also want to “map” your local filesystem locations.

You find the configuration for your research environment below:

Step 3. Getting started with iCommands

After installing and configuring the iCommands, you can log in to the Yoda environment using the iinit command. Note that the password you must use for the network disk is a data access password. Read more about using data access passwords. 

You can find examples of how to use the iCommands to transfer and manage files in sections 5.3, 5.4 and 5.5 of the iRODS beginner training.

You will find extra information in the iCommands manual here.