Getting Started

These tutorials show the capabilities of Distem by putting the user in two real situations. The experiments are supposed to be launched on the Grid’5000 platform in a Debian based environment.

As you probably read in the Distem presentation, the software is working following a server-client architecture. The server is based on the REST technology. To interact with the Distem server, different methods are available:

  1. Using the command-line executable
  2. Using the ruby client library
  3. Contacting the REST API directly
  4. Using a configuration file (XML)

In this tutorial we will present the usage of the first two methods. The documentation on the usage of this different methods can be found on the documentation page, further information is available on the FAQ page. At any moment if you have a question, find a bug or just want to give us some advice, you can contact the Distem team throught the mailing list.

1 Using Distem

1.1 Make a reservation

To start an experiment, you have to use some physical nodes on the testbed. As Distem requires some administrator privileges, the nodes must be reserved in deployment mode. Furthermore, on Grid’5000 some IP ranges are dedicated to addresses for virtual machines. However, these ranges must be reserved with g5k-subnets to be used. Throughout the tutorial the values that have to be replaced are written in upper case.

To perform this tutorial, you can reserve your nodes with the following command:

    frontend> oarsub -t deploy -l slash_22=1+nodes=NUMBER_OF_PHYSICAL_MACHINES, \
              walltime=TIME_OF_THE_RESERVATION -I
  • NUMBER_OF_PHYSICAL_MACHINES is the number of physical machines you want to get in your reservation, their address are stored in the $OAR_NODE_FILE environment variable.
  • TIME_OF_THE_RESERVATION Duration of the reservation, number of hour or time in HH:MM:SS format.

Once the reservation is performed, you can get the range of virtual IPs associated to your reservation with:

    frontend> g5k-subnets -sp

You will need this address for your experiments.

1.2 Preparing physical machines

Now that you got your reservation, the next step is to install an operating system on your allocated physical machines. You can deploy a Debian/Stretch environment -with NFS support, useful not to have to copy Distem filesystem images- using:

    frontend> kadeploy3 -f $OAR_NODE_FILE -e debian9-x64-nfs -k

Now, if all goes well, the nodes are ready to for the next step.

1.3 Distem Installation

Distem can easily be installed on the Grid’5000 nodes thanks to a help script called distem-bootstrap. This script is designed to install the Distem debian package and all its required dependencies.

distem-bootstrap can be launched as follows:

    frontend> distem-bootstrap

With Debian Buster you have to specify the Debian version:

    frontend> distem-bootstrap --debian-version buster

In addition to installing Distem on the physical nodes, distem-bootstrap also starts a coordinator daemon on the first node (griffon-1 for example) and initializes the nodes involved. The coordinator is used to perform all the operations on Distem, whatever the node targeted by the operation.

2 Network experiment: SCP VS RSync

This first tutorial aims at becoming familiar with Distem. The final goal is to compare the efficiency of SCP and RSync when transferring several files over the network. We will see that the behavior of each tool differs upon different network latency values.

This tutorial is split in 2 steps :

  1. Simple experiment using the command-line tool
  2. Scripted experiment using the ruby library

To start this experiment, you have to use two physical nodes on the testbed, please take a look at Make a reservation, Preparing physical machines and Distem Installation to get and configure your physical machines.

For the tutorial, we assume that the reserved nodes are griffon-1 and griffon-2.

2.1 Simple shell experiment

2.1.1 Platform setup

This step is dedicated to the emulated platform configuration. The following operations must be executed on the coordinator node, with the root user.

    frontend> ssh root@griffon-1

First of all, we must create a virtual network. In this tutorial, the virtual network will be named vnetwork and its address corresponds to the output of the g5k-subnets -sp command, previously executed on the frontend. We assume that the network address is

    coord> distem --create-vnetwork vnetwork=vnetwork,address=

Then, we must create the virtual nodes. We will create one node on each physical node, called node-1 and node-2. To create a virtual node, you must also provide a root file-system. Finally, to allow a password-less connection between the nodes, you can specify a pair of password-less ssh keys. In the example we use the following file (on the Nancy Grid’5000 site): /home/amerlin/public/distem/distem-fs-jessie.tar.gz.

First you have to download the image in your home, in our example, in the distem_img folder:

    frontend> wget 'http://public.nancy.grid5000.fr/~amerlin/distem/distem-fs-jessie.tar.gz' -P ~/distem_img

For the rest of the tutorial, we assume the image are in /home/USER/distem_img/. Replace USER by your username.

Let’s create the virtual nodes:

    coord> distem --create-vnode vnode=node-1,pnode=griffon-1,\
    coord> distem --create-vnode vnode=node-2,pnode=griffon-2,\

Now we create the network interfaces on each virtual node:

    coord> distem --create-viface vnode=node-1,iface=if0,vnetwork=vnetwork
    coord> distem --create-viface vnode=node-2,iface=if0,vnetwork=vnetwork

Command output example:

[{"name" => "node-1",
  "filesystem" =>
   {"sharedpath" => nil,
    "vnode" => "node-1",
    "shared" => nil,
    "path" => "/tmp/distem/rootfs-unique/node-1",
    "image" => "file:///home/USER/distem_img/distem-fs-jessie.tar.gz"},
  "id" => "0",
  "vifaces" =>
   [{"name" => "if0",
     "address" => "",
     "vinput" => nil,
     "voutput" => nil,
     "vnode" => "node-1",
     "vnetwork" => "vnetwork",
     "id" => "0"}],
  "host" => "",
  "gateway" => false,
  "status" => "INIT",
  "vcpu" => nil},
 {"name" => "node-2",
  "filesystem" =>
   {"sharedpath" => nil,
    "vnode" => "node-2",
    "shared" => nil,
    "path" => "/tmp/distem/rootfs-unique/node-2",
    "image" => "file:///home/USER/distem_img/distem-fs-jessie.tar.gz""},
  "id" => "1",
  "vifaces" =>
   [{"name" => "if0",
     "address" => "",
     "vinput" => nil,
     "voutput" => nil,
     "vnode" => "node-2",
     "vnetwork" => "vnetwork",
     "id" => "-1"}],
  "host" => "",
  "gateway" => false,
  "status" => "INIT",
  "vcpu" => nil}]

In particular you can note for each node the assigned IP address, in our example:

  • node-1:
  • node-2:

Finally, we start the virtual nodes:

    coord> distem --start-vnode node-1
    coord> distem --start-vnode node-2

At this time, you can connect to your nodes and perform your experiment. Either you can connect in your nodes, or you can run a command on a node:

  • connection in a node to get a shell (user:root, password:root):

    coord> distem --shell node-1

    This option uses lxc-console, so the exit key sequence (ctrl-a + q by default) can be modified if you are running inside a screen session.

  • execution of a command:

    coord> distem --execute vnode=node-1,command="hostname"

2.1.2 Experiment

We assume that the IP of the virtual nodes are:

  • node-1:
  • node-2:

To run the experiment we connect on the first node:

    coord> distem --shell node-1

We create 100 files of 50KB on the first node. This set of commands is performed inside the first node:

    node-1> mkdir /tmp/src
    node-1> for i in `seq 1 100`; do dd if=/dev/zero of=/tmp/src/$i bs=1K count=50; done

Still inside the first node, here is the core experiment (transfer of the 100 files with scp and rsync from node-1 to node-2):

    node-1> ssh "rm -rf /tmp/dst"
    node-1> time scp -rq /tmp/src
    node-1> ssh "rm -rf /tmp/dst"
    node-1> time rsync -r /tmp/src

Here, you can compare the time of execution for scp and rsync.

Now, we will modify the latency of the network links. In this experiment, we want to set a limitation on the interface of each virtual nodes (in output for each interface) in order to simulate a latency of 20ms on the network link. From the coordinator, let’s run:

    coord> distem --config-viface vnode=node-1,iface=if0,latency=20ms,\
    coord> distem --config-viface vnode=node-2,iface=if0,latency=20ms,\

Once the new latency is configured, you can run again the core experiment and observe the time required to perform scp or rsync. You can try to repeat the experiment by setting the link latency to 40, 60, 80 and 100ms.

2.2 Scripted experiment

All the operations performed before can be scripted in Ruby since Distem provide a Ruby API. Distem also provides users with a REST interface but this is out of the scope of this tutorial.

2.2.1 Platform setup

After having installed Distem with:

    frontend> distem-bootstrap --node-list $OAR_NODE_FILE

you can deploy the virtual platform by running a platform script on the coordinator:

    coord> ruby platform_setup.rb ""

platform_setup.rb is a script that:

  1. takes the virtual network address assigned by the OAR scheduler as a parameter ;
  2. builds the virtual network ;
  3. builds the virtual nodes node-1 and node-2,and their network interfaces.

Here is the source of this script:

# Import the Distem module
require 'distem'
# The path to the compressed filesystem image
# We can point to local file since our homedir is available from NFS
# Put the physical machines that have been assigned to you
# You can get that by executing: cat $OAR_NODE_FILE | uniq
pnodes=["pnode1","pnode2", ... ]
raise 'This experiment requires at least two physical machines' unless pnodes.size >= 2
# The first argument of the script is the address (in CIDR format)
# of the virtual network to set-up in our platform
# This ruby hash table describes our virtual network
vnet = {
  'name' => 'testnet',
  'address' => ARGV[0]
nodelist = ['node-1','node-2']
# Read SSH keys
private_key = IO.readlines('/root/.ssh/id_rsa').join
public_key = IO.readlines('/root/.ssh/id_rsa.pub').join
sshkeys = {
  'private' => private_key,
  'public' => public_key
# Connect to the Distem server (on http://localhost:4567 by default)
Distem.client do |cl|
  puts 'Creating virtual network'
  # Start by creating the virtual network
  cl.vnetwork_create(vnet['name'], vnet['address'])
  # Creating one virtual node per physical one
  puts 'Creating virtual nodes'
  # Create the first virtual node and set it to be hosted on
  # the first physical machine
  cl.vnode_create(nodelist[0], { 'host' => pnodes[0] }, sshkeys)
  # Specify the path to the compressed filesystem image
  # of this virtual node
  cl.vfilesystem_create(nodelist[0], { 'image' => FSIMG })
  # Create a virtual network interface and connect it to vnet
  cl.viface_create(nodelist[0], 'if0', { 'vnetwork' => vnet['name'], 'default' => 'true' })
  # Create the first virtual node and set it to be hosted on
  # the second physical machine
  cl.vnode_create(nodelist[1], { 'host' => pnodes[1] }, sshkeys)
  cl.vfilesystem_create(nodelist[1], { 'image' => FSIMG })
  cl.viface_create(nodelist[1], 'if0', { 'vnetwork' => vnet['name'] })
  puts 'Starting virtual nodes'
  # Starting the virtual nodes using the synchronous method
  nodelist.each do |nodename|

As a result, two virtual nodes are created and started on our virtual platform and connected to the same virtual network

2.2.2 Experiment

The experiment setup can be launched from root on the coordinator node with the following command:

    coord> ruby experiment.rb

Here is the code of experiment.rb:

require 'distem'
# Function that perform the calculation of the average
# of an array of values
def average(values)
  sum = values.inject(0){ |tmpsum,v| tmpsum + v.to_f }
  return sum / values.size
# Function that perform the calculation of the standard deviation
# of an array of values
def stddev(values,avg = nil)
  avg = average(values) unless avg
  sum = values.inject(0){ |tmpsum,v| tmpsum + ((v.to_f-avg) ** 2) }
  return Math.sqrt(sum / values.size)
# Describing the resources we are working with
ifname = 'if0'
node1 = {
  'name' => 'node-1',
  'address' => nil
node2 = {
  'name' => 'node-2',
  'address' => nil
# The parameters of our experimentation
latencies = ['0ms', '20ms', '40ms', '60ms']
results = {
  'scp' => {},
  'rsync' => {}
iterations = 5
Distem.client do |cl|
  # Getting the -automatically affected- address of each virtual nodes
  # virtual network interfaces
  node1['address'] = cl.viface_info(node1['name'],ifname)['address'].split('/')[0]
  node2['address'] = cl.viface_info(node2['name'],ifname)['address'].split('/')[0]
  # Creating the files we will use in our experimentation
    'mkdir -p /tmp/src ; cd /tmp/src ; \
     for i in `seq 1 100`; do \
      dd if=/dev/zero of=$i bs=1K count=50; \
  # Printing the current latency
  start_time = Time.now.to_f
  cl.vnode_execute(node1['name'], 'hostname')
  puts "Latency without any limitations #{Time.now.to_f - start_time}"
  # Preparing the description structure that will be used to
  # update virtual network interfaces latency
  desc = {
    'output' => {
      'latency' => {
        'delay' => nil
  # Starting our experiment for each specified latencies
  puts 'Starting tests'
  latencies.each do |latency|
    puts "Latency #{latency}"
    results['scp'][latency] = []
    results['rsync'][latency] = []
    # Update the latency description on virtual nodes
    desc['output']['latency']['delay'] = latency
    iterations.times do |iter|
      puts "\tIteration ##{iter}"
      # Launch SCP test
      # Cleaning target directory on node2
      cl.vnode_execute(node2['name'], 'rm -rf /tmp/dst')
      # Starting the copy from node1 to node2
      start_time = Time.now.to_f
        "scp -rq /tmp/src #{node2['address']}:/tmp/dst"
      results['scp'][latency] << Time.now - start_time
      # Launch RSYNC test
      # Cleaning target directory on node2
      cl.vnode_execute(node2['name'], 'rm -rf /tmp/dst')
      # Starting the copy from node1 to node2
      start_time = Time.now
        "rsync -r /tmp/src #{node2['address']}:/tmp/dst"
      results['rsync'][latency] << Time.now - start_time
puts "Rsync results:"
results['rsync'].keys.sort {|a,b| a.to_i <=> b.to_i}.each do |latency|
  values = results['rsync'][latency]
  avg = average(values)
  puts "\t#{latency}: [average=#{avg},standard_deviation=#{stddev(values,avg)}]"
puts "SCP results:"
results['scp'].keys.sort {|a,b| a.to_i <=> b.to_i}.each do |latency|
  values = results['scp'][latency]
  avg = average(values)
  puts "\t#{latency}: [average=#{avg},standard_deviation=#{stddev(values,avg)}]"

Now you are ready for some advanced experiments or for playing with SDN using Distem.