Tutorial
Getting Started
These tutorials show the capabilities of Distem by putting the user in two real situations. The experiments are supposed to be launched on the Grid’5000 platform in a Debian based environment.
As you probably read in the Distem presentation, the software is working following a server-client architecture. The server is based on the REST technology. To interact with the Distem server, different methods are available:
- Using the command-line executable
- Using the ruby client library
- Contacting the REST API directly
- Using a configuration file (XML)
In this tutorial we will present the usage of the first two methods. The documentation on the usage of this different methods can be found on the documentation page, further information is available on the FAQ page. At any moment if you have a question, find a bug or just want to give us some advice, you can contact the Distem team throught the mailing list.
1 Using Distem
1.1 Make a reservation
To start an experiment, you have to use some physical nodes on the testbed. As Distem requires some administrator privileges, the nodes must be reserved in deployment mode. Furthermore, on Grid’5000 some IP ranges are dedicated to addresses for virtual machines. However, these ranges must be reserved with g5k-subnets to be used. Throughout the tutorial the values that have to be replaced are written in upper case.
To perform this tutorial, you can reserve your nodes with the following command:
frontend> oarsub -t deploy -l slash_22=1+nodes=NUMBER_OF_PHYSICAL_MACHINES, \
walltime=TIME_OF_THE_RESERVATION -I- NUMBER_OF_PHYSICAL_MACHINES is the number of physical machines you want to get in your reservation, their address are stored in the $OAR_NODE_FILE environment variable.
- TIME_OF_THE_RESERVATION Duration of the reservation, number of hour or time in HH:MM:SS format.
Once the reservation is performed, you can get the range of virtual IPs associated to your reservation with:
You will need this address for your experiments.
1.2 Preparing physical machines
Now that you got your reservation, the next step is to install an operating system on your allocated physical machines. You can deploy a Debian/Stretch environment -with NFS support, useful not to have to copy Distem filesystem images- using:
Now, if all goes well, the nodes are ready to for the next step.
1.3 Distem Installation
Distem can easily be installed on the Grid’5000 nodes thanks to a help script called distem-bootstrap. This script is designed to install the Distem debian package and all its required dependencies.
distem-bootstrap can be launched as follows:
With Debian Buster you have to specify the Debian version:
In addition to installing Distem on the physical nodes, distem-bootstrap also starts a coordinator daemon on the first node (griffon-1 for example) and initializes the nodes involved. The coordinator is used to perform all the operations on Distem, whatever the node targeted by the operation.
2 Network experiment: SCP VS RSync
This first tutorial aims at becoming familiar with Distem. The final goal is to compare the efficiency of SCP and RSync when transferring several files over the network. We will see that the behavior of each tool differs upon different network latency values.
This tutorial is split in 2 steps :
- Simple experiment using the command-line tool
- Scripted experiment using the ruby library
To start this experiment, you have to use two physical nodes on the testbed, please take a look at Make a reservation, Preparing physical machines and Distem Installation to get and configure your physical machines.
For the tutorial, we assume that the reserved nodes are griffon-1 and griffon-2.
2.1 Simple shell experiment
2.1.1 Platform setup
This step is dedicated to the emulated platform configuration. The following operations must be executed on the coordinator node, with the root user.
First of all, we must create a virtual network. In this tutorial, the virtual network will be named vnetwork and its address corresponds to the output of the g5k-subnets -sp command, previously executed on the frontend. We assume that the network address is 10.144.0.0/22.
Then, we must create the virtual nodes. We will create one node on each physical node, called node-1 and node-2. To create a virtual node, you must also provide a root file-system. Finally, to allow a password-less connection between the nodes, you can specify a pair of password-less ssh keys. In the example we use the following file (on the Nancy Grid’5000 site): /home/amerlin/public/distem/distem-fs-jessie.tar.gz.
First you have to download the image in your home, in our example, in the distem_img folder:
frontend> wget 'http://public.nancy.grid5000.fr/~amerlin/distem/distem-fs-jessie.tar.gz' -P ~/distem_imgFor the rest of the tutorial, we assume the image are in /home/USER/distem_img/. Replace USER by your username.
Let’s create the virtual nodes:
coord> distem --create-vnode vnode=node-1,pnode=griffon-1,\
rootfs=file:///home/USER/distem_img/distem-fs-jessie.tar.gz,\
sshprivkey=/root/.ssh/id_rsa,sshpubkey=/root/.ssh/id_rsa.pub
coord> distem --create-vnode vnode=node-2,pnode=griffon-2,\
rootfs=file:///home/USER/distem_img/distem-fs-jessie.tar.gz,\
sshprivkey=/root/.ssh/id_rsa,sshpubkey=/root/.ssh/id_rsa.pubNow we create the network interfaces on each virtual node:
coord> distem --create-viface vnode=node-1,iface=if0,vnetwork=vnetwork
coord> distem --create-viface vnode=node-2,iface=if0,vnetwork=vnetworkCommand output example:
[{"name" => "node-1",
"filesystem" =>
{"sharedpath" => nil,
"vnode" => "node-1",
"shared" => nil,
"path" => "/tmp/distem/rootfs-unique/node-1",
"image" => "file:///home/USER/distem_img/distem-fs-jessie.tar.gz"},
"id" => "0",
"vifaces" =>
[{"name" => "if0",
"address" => "10.144.0.1/22",
"vinput" => nil,
"voutput" => nil,
"vnode" => "node-1",
"vnetwork" => "vnetwork",
"id" => "0"}],
"host" => "172.16.65.1",
"gateway" => false,
"status" => "INIT",
"vcpu" => nil},
{"name" => "node-2",
"filesystem" =>
{"sharedpath" => nil,
"vnode" => "node-2",
"shared" => nil,
"path" => "/tmp/distem/rootfs-unique/node-2",
"image" => "file:///home/USER/distem_img/distem-fs-jessie.tar.gz""},
"id" => "1",
"vifaces" =>
[{"name" => "if0",
"address" => "10.144.0.2/22",
"vinput" => nil,
"voutput" => nil,
"vnode" => "node-2",
"vnetwork" => "vnetwork",
"id" => "-1"}],
"host" => "172.16.65.2",
"gateway" => false,
"status" => "INIT",
"vcpu" => nil}]In particular you can note for each node the assigned IP address, in our example:
- node-1: 10.144.0.1/22
- node-2: 10.144.0.2/22
Finally, we start the virtual nodes:
At this time, you can connect to your nodes and perform your experiment. Either you can connect in your nodes, or you can run a command on a node:
connection in a node to get a shell (user:root, password:root):
This option uses lxc-console, so the exit key sequence (ctrl-a + q by default) can be modified if you are running inside a screen session.
execution of a command:
2.1.2 Experiment
We assume that the IP of the virtual nodes are:
- node-1: 10.144.0.1
- node-2: 10.144.0.2
To run the experiment we connect on the first node:
We create 100 files of 50KB on the first node. This set of commands is performed inside the first node:
node-1> mkdir /tmp/src
node-1> for i in `seq 1 100`; do dd if=/dev/zero of=/tmp/src/$i bs=1K count=50; doneStill inside the first node, here is the core experiment (transfer of the 100 files with scp and rsync from node-1 to node-2):
node-1> ssh 10.144.0.2 "rm -rf /tmp/dst"
node-1> time scp -rq /tmp/src 10.144.0.2:/tmp/dst
node-1> ssh 10.144.0.2 "rm -rf /tmp/dst"
node-1> time rsync -r /tmp/src 10.144.0.2:/tmp/dstHere, you can compare the time of execution for scp and rsync.
Now, we will modify the latency of the network links. In this experiment, we want to set a limitation on the interface of each virtual nodes (in output for each interface) in order to simulate a latency of 20ms on the network link. From the coordinator, let’s run:
coord> distem --config-viface vnode=node-1,iface=if0,latency=20ms,\
direction=OUTPUT
coord> distem --config-viface vnode=node-2,iface=if0,latency=20ms,\
direction=OUTPUTOnce the new latency is configured, you can run again the core experiment and observe the time required to perform scp or rsync. You can try to repeat the experiment by setting the link latency to 40, 60, 80 and 100ms.
2.2 Scripted experiment
All the operations performed before can be scripted in Ruby since Distem provide a Ruby API. Distem also provides users with a REST interface but this is out of the scope of this tutorial.
2.2.1 Platform setup
After having installed Distem with:
you can deploy the virtual platform by running a platform script on the coordinator:
platform_setup.rb is a script that:
- takes the virtual network address assigned by the OAR scheduler as a parameter ;
- builds the virtual network ;
- builds the virtual nodes node-1 and node-2,and their network interfaces.
Here is the source of this script:
#!/usr/bin/ruby
# Import the Distem module
require 'distem'
# The path to the compressed filesystem image
# We can point to local file since our homedir is available from NFS
FSIMG="file:///home/USER/distem_img/distem-fs-jessie.tar.gz""
# Put the physical machines that have been assigned to you
# You can get that by executing: cat $OAR_NODE_FILE | uniq
pnodes=["pnode1","pnode2", ... ]
raise 'This experiment requires at least two physical machines' unless pnodes.size >= 2
# The first argument of the script is the address (in CIDR format)
# of the virtual network to set-up in our platform
# This ruby hash table describes our virtual network
vnet = {
'name' => 'testnet',
'address' => ARGV[0]
}
nodelist = ['node-1','node-2']
# Read SSH keys
private_key = IO.readlines('/root/.ssh/id_rsa').join
public_key = IO.readlines('/root/.ssh/id_rsa.pub').join
sshkeys = {
'private' => private_key,
'public' => public_key
}
# Connect to the Distem server (on http://localhost:4567 by default)
Distem.client do |cl|
puts 'Creating virtual network'
# Start by creating the virtual network
cl.vnetwork_create(vnet['name'], vnet['address'])
# Creating one virtual node per physical one
puts 'Creating virtual nodes'
# Create the first virtual node and set it to be hosted on
# the first physical machine
cl.vnode_create(nodelist[0], { 'host' => pnodes[0] }, sshkeys)
# Specify the path to the compressed filesystem image
# of this virtual node
cl.vfilesystem_create(nodelist[0], { 'image' => FSIMG })
# Create a virtual network interface and connect it to vnet
cl.viface_create(nodelist[0], 'if0', { 'vnetwork' => vnet['name'], 'default' => 'true' })
# Create the first virtual node and set it to be hosted on
# the second physical machine
cl.vnode_create(nodelist[1], { 'host' => pnodes[1] }, sshkeys)
cl.vfilesystem_create(nodelist[1], { 'image' => FSIMG })
cl.viface_create(nodelist[1], 'if0', { 'vnetwork' => vnet['name'] })
puts 'Starting virtual nodes'
# Starting the virtual nodes using the synchronous method
nodelist.each do |nodename|
cl.vnode_start(nodename)
end
endAs a result, two virtual nodes are created and started on our virtual platform and connected to the same virtual network
2.2.2 Experiment
The experiment setup can be launched from root on the coordinator node with the following command:
Here is the code of experiment.rb:
#!/usr/bin/ruby
require 'distem'
# Function that perform the calculation of the average
# of an array of values
def average(values)
sum = values.inject(0){ |tmpsum,v| tmpsum + v.to_f }
return sum / values.size
end
# Function that perform the calculation of the standard deviation
# of an array of values
def stddev(values,avg = nil)
avg = average(values) unless avg
sum = values.inject(0){ |tmpsum,v| tmpsum + ((v.to_f-avg) ** 2) }
return Math.sqrt(sum / values.size)
end
# Describing the resources we are working with
ifname = 'if0'
node1 = {
'name' => 'node-1',
'address' => nil
}
node2 = {
'name' => 'node-2',
'address' => nil
}
# The parameters of our experimentation
latencies = ['0ms', '20ms', '40ms', '60ms']
results = {
'scp' => {},
'rsync' => {}
}
iterations = 5
Distem.client do |cl|
# Getting the -automatically affected- address of each virtual nodes
# virtual network interfaces
node1['address'] = cl.viface_info(node1['name'],ifname)['address'].split('/')[0]
node2['address'] = cl.viface_info(node2['name'],ifname)['address'].split('/')[0]
# Creating the files we will use in our experimentation
cl.vnode_execute(node1['name'],
'mkdir -p /tmp/src ; cd /tmp/src ; \
for i in `seq 1 100`; do \
dd if=/dev/zero of=$i bs=1K count=50; \
done'
)
# Printing the current latency
start_time = Time.now.to_f
cl.vnode_execute(node1['name'], 'hostname')
puts "Latency without any limitations #{Time.now.to_f - start_time}"
# Preparing the description structure that will be used to
# update virtual network interfaces latency
desc = {
'output' => {
'latency' => {
'delay' => nil
}
}
}
# Starting our experiment for each specified latencies
puts 'Starting tests'
latencies.each do |latency|
puts "Latency #{latency}"
results['scp'][latency] = []
results['rsync'][latency] = []
# Update the latency description on virtual nodes
desc['output']['latency']['delay'] = latency
cl.viface_update(node1['name'],ifname,desc)
cl.viface_update(node2['name'],ifname,desc)
iterations.times do |iter|
puts "\tIteration ##{iter}"
# Launch SCP test
# Cleaning target directory on node2
cl.vnode_execute(node2['name'], 'rm -rf /tmp/dst')
# Starting the copy from node1 to node2
start_time = Time.now.to_f
cl.vnode_execute(node1['name'],
"scp -rq /tmp/src #{node2['address']}:/tmp/dst"
)
results['scp'][latency] << Time.now - start_time
# Launch RSYNC test
# Cleaning target directory on node2
cl.vnode_execute(node2['name'], 'rm -rf /tmp/dst')
# Starting the copy from node1 to node2
start_time = Time.now
cl.vnode_execute('node-1',
"rsync -r /tmp/src #{node2['address']}:/tmp/dst"
)
results['rsync'][latency] << Time.now - start_time
end
end
end
puts "Rsync results:"
results['rsync'].keys.sort {|a,b| a.to_i <=> b.to_i}.each do |latency|
values = results['rsync'][latency]
avg = average(values)
puts "\t#{latency}: [average=#{avg},standard_deviation=#{stddev(values,avg)}]"
end
puts "SCP results:"
results['scp'].keys.sort {|a,b| a.to_i <=> b.to_i}.each do |latency|
values = results['scp'][latency]
avg = average(values)
puts "\t#{latency}: [average=#{avg},standard_deviation=#{stddev(values,avg)}]"
endNow you are ready for some advanced experiments or for playing with SDN using Distem.