Class: Distem::Daemon::DistemPnode
- Inherits:
-
Object
- Object
- Distem::Daemon::DistemPnode
- Defined in:
- lib/distem/daemon/distempnode.rb
Constant Summary
- @@lockslock =
Mutex.new
- @@locks =
{ :vnetsync => {}, }
- @@threads =
{ :pnode_init => {}, }
Instance Attribute Summary (collapse)
-
- (Object) default_network_gw
readonly
Returns the value of attribute default_network_gw.
-
- (Object) default_network_interface
readonly
Returns the value of attribute default_network_interface.
-
- (Object) linux_bridges
readonly
Hash = bridge_name.
-
- (Object) node_config
readonly
The NodeConfig object that allows to apply virtual resources specifications on physical nodes.
Instance Method Summary (collapse)
- - (Object) DELETE name
- - (Object) get_default_iface_from_vnode(vnode)
-
- (DistemPnode) initialize
constructor
A new instance of DistemPnode.
-
- (Object) pcpu_get(hostname, raising = true)
Get the description of the cpu of a physical node.
-
- (Object) pmemory_get(hostname, raising = true)
Get the description of the memory of a physical node.
-
- (Object) pnode_create(target, desc = {}, async = false)
Initialise a physical machine (launching daemon, creating cgroups, …) This step have to be performed to be able to create virtual nodes on a machine ==== Attributes *
target
the name/address of the physical machine *properties
async,max_vifaces,cpu_algorithm ==== Returns Resource::PNode object ==== Exceptions. -
- (Object) pnode_get(hostname, raising = true)
Get the description of a virtual node.
-
- (Object) pnode_quit(target)
Quit distem on a physical machine (remove everything that was created) ==== Returns Resource::PNode object ==== Exceptions.
-
- (Object) pnode_update(target, desc)
Update PNode properties.
-
- (Object) pnodes_delete_probes
Delete the probes on every PNode.
-
- (Object) pnodes_get_probes_data
Get the data generated by the probes.
-
- (Object) pnodes_launch_probes(ops, ref_time)
Launch some probes on every PNode.
-
- (Object) pnodes_restart_probes
Restart the probes on every PNode.
-
- (Object) pnodes_stop_probes
Stop the probes on every PNode.
- - (Object) set_global_arptable(compressed_data, arp_file)
- - (Object) set_global_etchosts(compressed_data)
- - (Object) set_peers_latencies(vnodes, matrix)
-
- (Object) vcpu_create(vnodename, desc)
Create a new virtual cpu on the targeted virtual node.
- - (Object) vcpu_get(vnodename)
- - (Object) vcpu_remove(vnodename)
- - (Object) vcpu_update(vnodename, desc)
- - (Object) vfilesystem_create(vnodename, desc)
-
- (Object) vfilesystem_image(vnodename)
Get a compressed archive of the current filesystem (tgz) WARNING: You have to contact the physical node the vnode is hosted on directly ==== Returns String object that describes the path to the archive ==== Exceptions.
-
- (Object) viface_attach(vnodename, vifacename, desc)
Connect a virtual node on a virtual network specifying which of it's virtual interface to use The IP address is auto assigned to the virtual interface Dettach the virtual interface if properties is empty You can change the traffic specification on the fly, only specifying a vtraffic property ==== Attributes *
vnodename
The VNode name (String) *vifacename
The VIface name (String) *properties
the address or the vnetwork to connect the virtual interface with (JSON, 'address' or 'vnetwork'), the traffic the interface will have to emulate (not mandatory, JSON, 'vtraffic', INPUT/OUTPUT) == Usage ==== Returns Resource::VIface object ==== Exceptions. -
- (Object) viface_create(vnodename, vifacename, desc)
Create a new virtual interface on the targeted virtual node (without attaching it to any network -> no ip address) ==== Attributes *
name
the name of the virtual interface (need to be unique on this virtual node) ==== Returns Resource::VIface object ==== Exceptions. -
- (Object) viface_detach(vnodename, vifacename)
Disconnect a virtual network interface from every networks it's connected to ==== Attributes *
vnodename
The VNode name (String) *vifacename
The VIface name (String) ==== Returns Resource::PNode object ==== Exceptions. -
- (Object) viface_get(vnodename, vifacename, raising = true)
Get the description of a virtual network interface ==== Returns Resource::VIface object ==== Exceptions.
-
- (Object) viface_remove(vnodename, vifacename)
Remove the virtual interface ==== Returns Resource::VIface object ==== Exceptions.
- - (Object) viface_update(vnodename, vifacename, desc)
- - (Object) vinput_get(vnodename, vifacename, raising = true)
- - (Object) vinput_update(vnodename, vifacename, desc)
-
- (Object) vnetwork_create(name, address, opts)
Create a new virtual network specifying his range of IP address (IPv4 atm).
-
- (Object) vnetwork_get(name, raising = true)
Get the description of a virtual network ==== Returns Resource::VNetwork object ==== Exceptions.
-
- (Object) vnetwork_remove(name)
Delete the virtual network ==== Returns Resource::VNetwork object ==== Exceptions.
-
- (Object) vnetworks_get
Get the list of the the currently created virtual networks ==== Returns Array of Resource::VNetwork objects ==== Exceptions.
-
- (Object) vnetworks_remove
Delete every virtual networks ==== Returns Array of Resource::VNetwork objects ==== Exceptions.
- - (Object) vnode_attach(name, host)
-
- (Object) vnode_create(names, descs, ssh_key = {}, async = false)
Create virtual nodes using a compressed file system image.
-
- (Object) vnode_get(name, raising = true)
Get the description of a virtual node ==== Returns Resource::VNode object ==== Exceptions.
-
- (Object) vnode_mode_update(name, mode)
Change the mode of a virtual node (normal or gateway) ==== Attributes *
mode
“Normal” or “Gateway” ==== Returns Resource::VNode object ==== Exceptions. -
- (Object) vnode_remove(name)
Remove the virtual node (“Cascade” removing -> remove all the vroutes it apears as gateway) ==== Returns Resource::VNode object ==== Exceptions.
-
- (Object) vnode_start(name, async = false)
Same as vnode_set_status(name,Resource::Status::RUNNING,properties).
-
- (Object) vnode_status_update(name, status, async = false)
Change the status of the -previously created- virtual node.
-
- (Object) vnode_stop(name, async = false)
Same as vnode_set_status(name,Resource::Status::DOWN,properties).
-
- (Object) vnode_update(names, descs, async = false)
Update the vnode resource.
- - (Object) vnodes_execute(names, command)
- - (Object) vnodes_freeze(names, async)
-
- (Object) vnodes_get
Get the list of the the currently created virtual nodes ==== Returns Array of Resource::PNode objects ==== Exceptions.
-
- (Object) vnodes_remove(names)
Remove the given virtual nodes, or every if names is nil ==== Returns Array of Resource::PNode objects ==== Exceptions.
- - (Object) vnodes_stop(names, async = false)
- - (Object) vnodes_unfreeze(names, async)
- - (Object) voutput_get(vnodename, vifacename, raising = true)
- - (Object) voutput_update(vnodename, vifacename, desc)
-
- (Object) vroute_create(networksrc, networkdst, nodegw)
Create a virtual route (“go from to via ”).
-
- (Object) vtraffic_update(vnodename, vifacename, desc)
Configure the virtual traffic on a virtual network interface, replacing previous values ==== Attributes *
vnodename
The VNode name (String) *vifacename
The VIface name (String) *desc
Hash that represents the VTraffic description (see Lib::Validator) ==== Returns Resource::VIface object ==== Exceptions. - - (Object) wait_vnodes(opts)
Constructor Details
- (DistemPnode) initialize
Returns a new instance of DistemPnode
31 32 33 34 35 36 37 38 39 40 41 42 |
# File 'lib/distem/daemon/distempnode.rb', line 31 def initialize() #Thread::abort_on_exception = true @node_name = Socket::gethostname @node_config = Node::ConfigManager.new @collector = nil @etchosts_updated = nil @linux_bridges = {} @routing_interfaces = {} @vnetworks_linked_to_bridge = {} @default_network_interface = Lib::NetTools.get_default_iface @default_network_gw = Lib::NetTools.get_default_gateway end |
Instance Attribute Details
- (Object) default_network_gw (readonly)
Returns the value of attribute default_network_gw
29 30 31 |
# File 'lib/distem/daemon/distempnode.rb', line 29 def default_network_gw @default_network_gw end |
- (Object) default_network_interface (readonly)
Returns the value of attribute default_network_interface
28 29 30 |
# File 'lib/distem/daemon/distempnode.rb', line 28 def default_network_interface @default_network_interface end |
- (Object) linux_bridges (readonly)
Hash = bridge_name
27 28 29 |
# File 'lib/distem/daemon/distempnode.rb', line 27 def linux_bridges @linux_bridges end |
- (Object) node_config (readonly)
The NodeConfig object that allows to apply virtual resources specifications on physical nodes
15 16 17 |
# File 'lib/distem/daemon/distempnode.rb', line 15 def node_config @node_config end |
Instance Method Details
- (Object) DELETE name
897 |
# File 'lib/distem/daemon/distempnode.rb', line 897 @vnetworks_linked_to_bridge[brname].delete(name) |
- (Object) get_default_iface_from_vnode(vnode)
1038 1039 1040 1041 1042 1043 |
# File 'lib/distem/daemon/distempnode.rb', line 1038 def get_default_iface_from_vnode(vnode) # try to find an interface in the administration network since it is # supposed to be reachable in any case ifadm = vnode.vifaces.select { |viface| viface.name == 'ifadm' } return ifadm.empty? ? vnode.vifaces[0] : ifadm[0] end |
- (Object) pcpu_get(hostname, raising = true)
Get the description of the cpu of a physical node
169 170 171 172 |
# File 'lib/distem/daemon/distempnode.rb', line 169 def pcpu_get(hostname, raising = true) pnode = pnode_get(hostname) return pnode.cpu end |
- (Object) pmemory_get(hostname, raising = true)
Get the description of the memory of a physical node
175 176 177 178 |
# File 'lib/distem/daemon/distempnode.rb', line 175 def pmemory_get(hostname, raising = true) pnode = pnode_get(hostname) return pnode.memory end |
- (Object) pnode_create(target, desc = {}, async = false)
Initialise a physical machine (launching daemon, creating cgroups, …) This step have to be performed to be able to create virtual nodes on a machine
Attributes
-
target
the name/address of the physical machine -
properties
async,max_vifaces,cpu_algorithm
Returns
Resource::PNode object
Exceptions
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
# File 'lib/distem/daemon/distempnode.rb', line 54 def pnode_create(target,desc={},async=false) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(target) begin async = parse_bool(async) pnode = @node_config.pnode Node::Admin.init_node(pnode,desc) @node_config.vplatform.add_pnode(pnode) pnode.status = Resource::Status::RUNNING pnode_update(pnode.address.to_s,desc) return pnode rescue Lib::AlreadyExistingResourceError raise rescue Exception destroy(pnode) if pnode raise end end |
- (Object) pnode_get(hostname, raising = true)
Get the description of a virtual node
Attributes
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# File 'lib/distem/daemon/distempnode.rb', line 122 def pnode_get(hostname, raising = true) pnode = nil if hostname and Lib::NetTools.localaddr?(hostname) pnode = @node_config.pnode else hostname = '' unless hostname begin address = Resolv.getaddress(hostname) rescue Resolv::ResolvError raise Lib::InvalidParameterError, hostname end pnode = @node_config.vplatform.get_pnode_by_address(address) end raise Lib::ResourceNotFoundError, hostname if raising and !pnode return pnode end |
- (Object) pnode_quit(target)
Quit distem on a physical machine (remove everything that was created)
Returns
Resource::PNode object
Exceptions
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
# File 'lib/distem/daemon/distempnode.rb', line 94 def pnode_quit(target) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(target) pnode = pnode_get(target,false) pnode.status = Resource::Status::CONFIGURING vnetworks_remove() pnodes_delete_probes() if @collector if @etchosts_updated tmp = Tempfile.new("distem_etc_hosts") IO.readlines('/etc/hosts').each { |line| if !@etchosts_updated.include?(line) tmp.write(line) end } tmp.close FileUtils.mv(tmp.path, '/etc/hosts') File.chmod(0644, '/etc/hosts') tmp.unlink end Lib::FileManager::clean_cache Node::Admin.quit_node() pnode.status = Resource::Status::READY return pnode end |
- (Object) pnode_update(target, desc)
Update PNode properties
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
# File 'lib/distem/daemon/distempnode.rb', line 73 def pnode_update(target,desc) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(target) pnode = pnode_get(target) if desc['algorithms'] if desc['algorithms']['cpu'] algo = desc['algorithms']['cpu'].upcase raise InvalidParameterError "algorithms/cpu" unless \ [Algorithm::CPU::GOV.upcase, Algorithm::CPU::HOGS.upcase].include?(algo) pnode.algorithms[:cpu] = algo return {'algorithms' => {'cpu'=> algo }} end end end |
- (Object) pnodes_delete_probes
Delete the probes on every PNode
158 159 160 161 |
# File 'lib/distem/daemon/distempnode.rb', line 158 def pnodes_delete_probes() pnodes_stop_probes() @collector = nil end |
- (Object) pnodes_get_probes_data
Get the data generated by the probes
164 165 166 |
# File 'lib/distem/daemon/distempnode.rb', line 164 def pnodes_get_probes_data() return @collector.data end |
- (Object) pnodes_launch_probes(ops, ref_time)
Launch some probes on every PNode
142 143 144 145 |
# File 'lib/distem/daemon/distempnode.rb', line 142 def pnodes_launch_probes(ops, ref_time) @collector = DataCollection::Collector.new(ref_time.to_f, ops) @collector.run end |
- (Object) pnodes_restart_probes
Restart the probes on every PNode
148 149 150 |
# File 'lib/distem/daemon/distempnode.rb', line 148 def pnodes_restart_probes() @collector.restart if @collector end |
- (Object) pnodes_stop_probes
Stop the probes on every PNode
153 154 155 |
# File 'lib/distem/daemon/distempnode.rb', line 153 def pnodes_stop_probes() @collector.stop if @collector end |
- (Object) set_global_arptable(compressed_data, arp_file)
1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 |
# File 'lib/distem/daemon/distempnode.rb', line 1045 def set_global_arptable(compressed_data, arp_file) shared_fs = [] private_fs = [] @node_config.vplatform.vnodes.each_value { |vnode| if vnode.filesystem.shared shared_fs << vnode else private_fs << vnode end } data = Zlib::Inflate.inflate(Base64.decode64(compressed_data)) @node_config.set_global_arptable(shared_fs.first, data, arp_file) if !shared_fs.empty? private_fs.each {|vnode| @node_config.set_global_arptable(vnode, data, arp_file) } if !private_fs.empty? w = Distem::Lib::Synchronization::SlidingWindow.new(100) (shared_fs + private_fs).each { |vnode| w.add("ssh -q -o StrictHostKeyChecking=no #{get_default_iface_from_vnode(vnode).address.address.to_s} \"arp -f #{arp_file} 2>/dev/null\"") } w.run end |
- (Object) set_global_etchosts(compressed_data)
1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 |
# File 'lib/distem/daemon/distempnode.rb', line 1009 def set_global_etchosts(compressed_data) shared_fs = [] private_fs = [] @node_config.vplatform.vnodes.each_value { |vnode| if vnode.filesystem.shared shared_fs << vnode else private_fs << vnode end } data = Zlib::Inflate.inflate(Base64.decode64(compressed_data)) @node_config.set_global_etchosts(shared_fs.first, data) if !shared_fs.empty? private_fs.each {|vnode| @node_config.set_global_etchosts(vnode, data) } if !private_fs.empty? @etchosts_updated = [] File.open('/etc/hosts','a') {|f| f.puts("\n") data.split("\n").each { |l| line = "#{l}\n" if !@etchosts_updated.include?(line) @etchosts_updated << line f.write(line) end } } end |
- (Object) set_peers_latencies(vnodes, matrix)
992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 |
# File 'lib/distem/daemon/distempnode.rb', line 992 def set_peers_latencies(vnodes, matrix) tids = [] matrix.each_pair { |vnode_name,destinations| tids << Thread.new { vnode = vnode_get(vnode_name) vnode.vifaces[0].latency_filters = destinations if vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING @node_config.vnode_reconfigure(vnode) vnode.status = Resource::Status::RUNNING end } } tids.each { |tid| tid.join } return true end |
- (Object) vcpu_create(vnodename, desc)
Create a new virtual cpu on the targeted virtual node. By default all the virtual nodes on a same physical one are sharing available CPU resources, using this method you can allocate some cores to a virtual node and apply some limitations on them
Attributes
-
corenb
the number of cores to allocate (need to have enough free ones on the physical node) -
frequency
(optional) the frequency each node have to be set (need to be lesser or equal than the physical core frequency). If the frequency is included in ]0,1] it'll be interpreted as a percentage of the physical core frequency, otherwise the frequency will be set to the specified number
Returns
Resource::VCPU object
Exceptions
695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 |
# File 'lib/distem/daemon/distempnode.rb', line 695 def vcpu_create(vnodename,desc) begin vnode = vnode_get(vnodename) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) downkeys(desc) corenb = nil val = nil unit = 'mhz' if desc['vcores'] raise Lib::InvalidParameterError, 'vcores' unless desc['vcores'].is_a?(Hash) val = desc['vcores'].values[0]['frequency'] corenb = desc['vcores'].keys.length else if desc['val'] val = desc['val'] unit = desc['unit'] if desc.has_key?('unit') else val = 1 unit = 'ratio' end corenb = desc['corenb'].to_i || 1 end vnode.add_vcpu(corenb,val,unit) vnode.vcpu.attach if vnode.host if vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING @node_config.vnode_reconfigure(vnode) vnode.status = Resource::Status::RUNNING end return vnode.vcpu rescue Lib::AlreadyExistingResourceError raise rescue Exception vnode.remove_vcpu() if vnode raise end end |
- (Object) vcpu_get(vnodename)
739 740 741 742 743 744 745 |
# File 'lib/distem/daemon/distempnode.rb', line 739 def vcpu_get(vnodename) vnode = vnode_get(vnodename) raise Lib::UninitializedResourceError, 'vcpu' unless vnode.vcpu return vnode.vcpu end |
- (Object) vcpu_remove(vnodename)
787 788 789 790 791 792 793 794 795 796 |
# File 'lib/distem/daemon/distempnode.rb', line 787 def vcpu_remove(vnodename) vnode = vnode_get(vnodename) raise Lib::UninitializedResourceError, 'vcpu' unless vnode.vcpu vcpu = vnode.vcpu vnode.remove_vcpu() return vcpu end |
- (Object) vcpu_update(vnodename, desc)
747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 |
# File 'lib/distem/daemon/distempnode.rb', line 747 def vcpu_update(vnodename,desc) begin vnode = vnode_get(vnodename) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) vcpu = vcpu_get(vnode.name) downkeys(desc) val = nil unit = 'mhz' if desc['vcores'] raise Lib::InvalidParameterError, 'vcores' unless desc['vcores'].is_a?(Hash) val = desc['vcores']['0']['frequency'] else if desc['val'] val = desc['val'] unit = desc['unit'] if desc.has_key?('unit') else val = 1 unit = 'ratio' end end vcpu.update_vcores(val,unit) if vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING @node_config.vnode_reconfigure(vnode) vnode.status = Resource::Status::RUNNING end return vnode.vcpu rescue Lib::AlreadyExistingResourceError raise rescue Exception vnode.remove_vcpu() if vnode raise end end |
- (Object) vfilesystem_create(vnodename, desc)
799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 |
# File 'lib/distem/daemon/distempnode.rb', line 799 def vfilesystem_create(vnodename,desc) vnode = vnode_get(vnodename) raise Lib::AlreadyExistingResourceError, 'filesystem' if vnode.filesystem raise Lib::MissingParameterError, "filesystem/image" unless \ desc['image'] desc['shared'] = parse_bool(desc['shared']) desc['cow'] = parse_bool(desc['cow']) desc['disk_throttling'] = nil if !desc.has_key?('disk_throttling') vnode.filesystem = Resource::FileSystem.new(vnode,desc['image'],desc['shared'],desc['cow'],desc['disk_throttling']) return vnode.filesystem end |
- (Object) vfilesystem_image(vnodename)
Get a compressed archive of the current filesystem (tgz) WARNING: You have to contact the physical node the vnode is hosted on directly
Returns
String object that describes the path to the archive
Exceptions
821 822 823 824 825 826 827 828 829 830 831 |
# File 'lib/distem/daemon/distempnode.rb', line 821 def vfilesystem_image(vnodename) vnode = vnode_get(vnodename) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) archivepath = nil if vnode.filesystem.shared archivepath = Lib::FileManager::compress(vnode.filesystem.sharedpath) else archivepath = Lib::FileManager::compress(vnode.filesystem.path) end return archivepath end |
- (Object) viface_attach(vnodename, vifacename, desc)
Connect a virtual node on a virtual network specifying which of it's virtual interface to use The IP address is auto assigned to the virtual interface Dettach the virtual interface if properties is empty You can change the traffic specification on the fly, only specifying a vtraffic property
Attributes
-
vnodename
The VNode name (String) -
vifacename
The VIface name (String) -
properties
the address or the vnetwork to connect the virtual interface with (JSON, 'address' or 'vnetwork'), the traffic the interface will have to emulate (not mandatory, JSON, 'vtraffic', INPUT/OUTPUT)
Usage
Returns
Resource::VIface object
Exceptions
521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 |
# File 'lib/distem/daemon/distempnode.rb', line 521 def viface_attach(vnodename,vifacename,desc) begin vnode = vnode_get(vnodename) viface = viface_get(vnodename,vifacename) downkeys(desc) desc['vnetwork'] = desc['vnetwork'].gsub(' ','_') if desc['vnetwork'] raise Lib::MissingParameterError, "address&macaddress&vnetwork" if \ ((!desc['address'] or desc['address'].empty?) \ or (!desc['macaddress'] or desc['macaddress'].empty?) \ or (!desc['vnetwork'] or desc['vnetwork'].empty?)) vplatform = @node_config.vplatform viface.macaddress = desc['macaddress'] viface.bridge = desc['bridge'] if desc['address'] and !desc['address'].empty? begin address = IPAddress.parse(desc['address']) rescue ArgumentError raise Lib::InvalidParameterError, desc['address'] end prop = desc['address'] vnetwork = vplatform.get_vnetwork_by_address(prop) end if desc['vnetwork'] and !desc['vnetwork'].empty? prop = desc['vnetwork'] vnetwork = vplatform.get_vnetwork_by_name(prop) end raise Lib::ResourceNotFoundError, "vnetwork:#{prop}" unless vnetwork if desc['address'] vnetwork.add_vnode(vnode,viface,address) else vnetwork.add_vnode(vnode,viface) end desc.delete('address') desc.delete('vnetwork') vtraffic_update(vnode.name,viface.name,desc) unless desc.empty? return viface rescue Lib::AlreadyExistingResourceError raise rescue Exception vnetwork.remove_vnode(vnode) if vnetwork raise end end |
- (Object) viface_create(vnodename, vifacename, desc)
Create a new virtual interface on the targeted virtual node (without attaching it to any network -> no ip address)
Attributes
-
name
the name of the virtual interface (need to be unique on this virtual node)
Returns
Resource::VIface object
Exceptions
461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 |
# File 'lib/distem/daemon/distempnode.rb', line 461 def viface_create(vnodename,vifacename,desc) begin vifacename = vifacename.gsub(' ','_') vnode = vnode_get(vnodename) viface = Resource::VIface.new(vifacename,vnode) vnode.add_viface(viface) downkeys(desc) viface_update(vnode.name,viface.name,desc) return viface rescue Lib::AlreadyExistingResourceError raise rescue Exception vnode.remove_viface(viface) if vnode and viface raise end end |
- (Object) viface_detach(vnodename, vifacename)
Disconnect a virtual network interface from every networks it's connected to
Attributes
-
vnodename
The VNode name (String) -
vifacename
The VIface name (String)
Returns
Resource::PNode object
Exceptions
580 581 582 583 584 585 |
# File 'lib/distem/daemon/distempnode.rb', line 580 def viface_detach(vnodename,vifacename) viface = viface_get(vnodename,vifacename) viface.detach() return viface end |
- (Object) viface_get(vnodename, vifacename, raising = true)
Get the description of a virtual network interface
Returns
Resource::VIface object
Exceptions
592 593 594 595 596 597 598 599 600 |
# File 'lib/distem/daemon/distempnode.rb', line 592 def viface_get(vnodename,vifacename,raising = true) vifacename = vifacename.gsub(' ','_') vnode = vnode_get(vnodename,raising) viface = vnode.get_viface_by_name(vifacename) raise Lib::ResourceNotFoundError, vifacename if raising and !viface return viface end |
- (Object) viface_remove(vnodename, vifacename)
Remove the virtual interface
Returns
Resource::VIface object
Exceptions
483 484 485 486 487 488 489 490 |
# File 'lib/distem/daemon/distempnode.rb', line 483 def viface_remove(vnodename,vifacename) vnode = vnode_get(vnodename) viface = viface_get(vnodename,vifacename) viface_detach(vnode.name,viface.name) vnode.remove_viface(viface) return viface end |
- (Object) viface_update(vnodename, vifacename, desc)
492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 |
# File 'lib/distem/daemon/distempnode.rb', line 492 def viface_update(vnodename,vifacename,desc) ret = nil downkeys(desc) if !desc or desc.empty? ret = viface_detach(vnodename,vifacename) else if (desc.keys - ['input'] - ['output']).size == 0 #only vtraffic change ret = vtraffic_update(vnodename,vifacename,desc) else ret = viface_attach(vnodename,vifacename,desc) end end return ret end |
- (Object) vinput_get(vnodename, vifacename, raising = true)
623 624 625 626 627 628 629 |
# File 'lib/distem/daemon/distempnode.rb', line 623 def vinput_get(vnodename,vifacename, raising = true) viface = viface_get(vnodename,vifacename) raise Lib::UninitializedResourceError, 'input' if raising and !viface.vinput return viface.vinput end |
- (Object) vinput_update(vnodename, vifacename, desc)
631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 |
# File 'lib/distem/daemon/distempnode.rb', line 631 def vinput_update(vnodename,vifacename,desc) vnode = vnode_get(vnodename) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) viface = viface_get(vnodename,vifacename) downkeys(desc) if desc and !desc.empty? viface.vinput = Resource::VIface::VTraffic.new(viface, Resource::VIface::VTraffic::Direction::INPUT,desc) else viface.vinput = nil end if vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING @node_config.vnode_reconfigure(vnode) vnode.status = Resource::Status::RUNNING end return viface.vinput end |
- (Object) vnetwork_create(name, address, opts)
Create a new virtual network specifying his range of IP address (IPv4 atm).
Attributes
-
name
the -unique- name of the virtual network (it will be used in a lot of methods) -
address
the address in the CIDR (10.0.0.1/24) or IP/NetMask (10.0.0.1/255.255.255.0) format -
opts
options that contains the number of pnodes and the vxlan_id
Returns
Resource::VNetwork object
Exceptions
842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 |
# File 'lib/distem/daemon/distempnode.rb', line 842 def vnetwork_create(name,address,opts) begin name = name.gsub(' ','_') if name root_interface = opts.has_key?('root_interface') ? opts['root_interface'] : @default_network_interface vnetwork = Resource::VNetwork.new(address,name,opts['nb_pnodes'].to_i, opts) # Check if the linux bridge has to be created if !@linux_bridges.has_key?(root_interface) brname = Lib::NetTools.set_bridge(root_interface, root_interface == @default_network_interface ? @default_network_gw : nil) if brname @linux_bridges[root_interface] = brname else raise Lib::AlreadyExistingResourceError end @vnetworks_linked_to_bridge[@linux_bridges[root_interface]] = [] end @vnetworks_linked_to_bridge[@linux_bridges[root_interface]] << name case opts['network_type'] when 'vxlan' address = IPAddress::IPv4::parse_u32(vnetwork.address.last.to_u32 - opts['pnode_index']).to_s netmask = vnetwork.address.netmask Lib::NetTools.create_vxlan_interface(opts['vxlan_id'], opts['vxlan_mcast_id'], address, netmask, @linux_bridges[root_interface]) @routing_interfaces[name] = Lib::NetTools.set_new_nic(address, netmask, Lib::NetTools::VXLAN_BRIDGE_PREFIX + opts['vxlan_id'].to_s) when 'classical' @routing_interfaces[name] = Lib::NetTools.set_new_nic( IPAddress::IPv4::parse_u32(vnetwork.address.last.to_u32 - opts['pnode_index']).to_s, vnetwork.address.netmask, @linux_bridges[root_interface]) else raise end @node_config.vnetwork_add(vnetwork) return vnetwork rescue Lib::AlreadyExistingResourceError raise rescue Exception destroy(vnetwork) if vnetwork raise end end |
- (Object) vnetwork_get(name, raising = true)
Get the description of a virtual network
Returns
Resource::VNetwork object
Exceptions
933 934 935 936 937 938 |
# File 'lib/distem/daemon/distempnode.rb', line 933 def vnetwork_get(name,raising = true) name = name.gsub(' ','_') vnetwork = @node_config.vplatform.get_vnetwork_by_name(name) raise Lib::ResourceNotFoundError, name if raising and !vnetwork return vnetwork end |
- (Object) vnetwork_remove(name)
Delete the virtual network
Returns
Resource::VNetwork object
Exceptions
891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 |
# File 'lib/distem/daemon/distempnode.rb', line 891 def vnetwork_remove(name) vnetwork = vnetwork_get(name) vnetwork.vnodes.each_pair do |vnode,viface| viface_detach(vnode.name,viface.name) end Lib::NetTools.unset_nic(@routing_interfaces[name]) @routing_interfaces.delete(name) if vnetwork.opts['network_type'] == 'vxlan' Lib::NetTools.remove_vxlan_interface(vnetwork.opts['vxlan_id']) end root_interface = vnetwork.opts.has_key?('root_interface') ? vnetwork.opts['root_interface'] : @default_network_interface brname = @linux_bridges[root_interface] @vnetworks_linked_to_bridge[brname].delete(name) if @vnetworks_linked_to_bridge[brname].empty? Lib::NetTools.unset_bridge(brname, root_interface == @default_network_interface ? @default_network_gw : nil) @vnetworks_linked_to_bridge.delete(brname) @linux_bridges.delete(vnetwork.opts['root_interface']) end @node_config.vnetwork_remove(vnetwork) return vnetwork end |
- (Object) vnetworks_get
Get the list of the the currently created virtual networks
Returns
Array of Resource::VNetwork objects
Exceptions
945 946 947 |
# File 'lib/distem/daemon/distempnode.rb', line 945 def vnetworks_get() return @node_config.vplatform.vnetworks end |
- (Object) vnetworks_remove
Delete every virtual networks
Returns
Array of Resource::VNetwork objects
Exceptions
918 919 920 921 922 923 924 925 926 |
# File 'lib/distem/daemon/distempnode.rb', line 918 def vnetworks_remove() vnetworks = nil vnetworks = @node_config.vplatform.vnetworks # Required to get the vnetworks before iterating over them since @node_config.vplatform.vnetworks is modified t = vnetworks.values t.each { |vnetwork| vnetwork_remove(vnetwork.name) } return vnetworks end |
- (Object) vnode_attach(name, host)
304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 |
# File 'lib/distem/daemon/distempnode.rb', line 304 def vnode_attach(name,host) vnode = vnode_get(name) pnode = @node_config.pnode if pnode raise Lib::UninitializedResourceError, pnode.address.to_s unless \ pnode.status == Resource::Status::RUNNING else raise Lib::ResourceNotFoundError host if host end if vnode.host and vnode.status == Resource::Status::RUNNING raise Lib::AlreadyExistingResourceError, 'host' else vnode.host = pnode end return vnode end |
- (Object) vnode_create(names, descs, ssh_key = {}, async = false)
Create virtual nodes using a compressed file system image.
Attributes
-
names
the -unique- name of the virtual nodes to create (it will be used in a lot of methods) -
properties
target,image,async,fs_shared,ssh_key
Returns
Resource::VNode object
Exceptions
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 |
# File 'lib/distem/daemon/distempnode.rb', line 189 def vnode_create(names,descs,ssh_key={},async=false) async = parse_bool(async) begin raise Lib::ArgumentMissingError "names" if !names names = [names] if !names.is_a?(Array) vnodes = [] names.each_index { |i| name = names[i] name = name.gsub(' ','_') desc = descs[i] downkeys(desc) vnode = Resource::VNode.new(name,ssh_key) @node_config.vnode_add(vnode) vnodes << vnode } vnode_update(names,descs,async) return vnodes rescue Lib::AlreadyExistingResourceError raise rescue Exception raise end end |
- (Object) vnode_get(name, raising = true)
Get the description of a virtual node
Returns
Resource::VNode object
Exceptions
326 327 328 329 330 331 |
# File 'lib/distem/daemon/distempnode.rb', line 326 def vnode_get(name, raising = true) name = name.gsub(' ','_') vnode = @node_config.get_vnode(name) raise Lib::ResourceNotFoundError, name if raising and !vnode return vnode end |
- (Object) vnode_mode_update(name, mode)
Change the mode of a virtual node (normal or gateway)
Attributes
-
mode
“Normal” or “Gateway”
Returns
Resource::VNode object
Exceptions
430 431 432 433 434 435 436 437 438 439 440 441 442 443 |
# File 'lib/distem/daemon/distempnode.rb', line 430 def vnode_mode_update(name,mode) vnode = vnode_get(name) case mode.upcase when Resource::VNode::MODE_GATEWAY.upcase vnode.gateway = true when Resource::VNode::MODE_NORMAL.upcase vnode.gateway = false else raise Lib::InvalidParameterError, "mode:#{mode}" end return vnode end |
- (Object) vnode_remove(name)
Remove the virtual node (“Cascade” removing -> remove all the vroutes it apears as gateway)
Returns
Resource::VNode object
Exceptions
275 276 277 278 279 280 281 282 283 284 |
# File 'lib/distem/daemon/distempnode.rb', line 275 def vnode_remove(name) vnode = vnode_get(name) raise Lib::BusyResourceError, "#{vnode.name}/running" if \ vnode.status == Resource::Status::RUNNING ret = vnode.dup vnode.vifaces.each { |viface| viface_remove(name,viface.name) } vnode.remove_vcpu() @node_config.vnode_remove(vnode) return ret end |
- (Object) vnode_start(name, async = false)
Same as vnode_set_status(name,Resource::Status::RUNNING,properties)
359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 |
# File 'lib/distem/daemon/distempnode.rb', line 359 def vnode_start(name,async=false) async = parse_bool(async) vnode = vnode_get(name) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) raise Lib::BusyResourceError, vnode.name if \ vnode.status == Resource::Status::CONFIGURING raise Lib::ResourceError, "#{vnode.name} already running" if \ vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING vnode.host = @node_config.pnode unless vnode.host vnode.vcpu.attach if vnode.vcpu and !vnode.vcpu.attached? @node_config.vnode_start(vnode, self) vnode.status = Resource::Status::RUNNING return [vnode] end |
- (Object) vnode_status_update(name, status, async = false)
Change the status of the -previously created- virtual node.
Attributes
-
status
the status to set: “Running”, “Ready”, or “Down” -
properties
async
Returns
Resource::VNode object
Exceptions
342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 |
# File 'lib/distem/daemon/distempnode.rb', line 342 def vnode_status_update(name,status,async=false) async = parse_bool(async) vnode = nil raise Lib::InvalidParameterError, status unless \ Resource::Status.valid?(status) if status.upcase == Resource::Status::RUNNING vnode = vnode_start(name,async) elsif status.upcase == Resource::Status::DOWN vnode = vnode_stop(name,async) else raise Lib::InvalidParameterError, status end return vnode end |
- (Object) vnode_stop(name, async = false)
Same as vnode_set_status(name,Resource::Status::DOWN,properties)
379 380 381 382 383 384 385 386 387 388 389 390 391 392 |
# File 'lib/distem/daemon/distempnode.rb', line 379 def vnode_stop(name, async=false) async = parse_bool(async) vnode = vnode_get(name) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) raise Lib::BusyResourceError, vnode.name if \ vnode.status == Resource::Status::CONFIGURING raise Lib::UninitializedResourceError, vnode.name if \ vnode.status == Resource::Status::INIT vnode.status = Resource::Status::CONFIGURING @node_config.vnode_stop(vnode) vnode.status = Resource::Status::DOWN return vnode end |
- (Object) vnode_update(names, descs, async = false)
Update the vnode resource
228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 |
# File 'lib/distem/daemon/distempnode.rb', line 228 def vnode_update(names,descs,async=false) async = parse_bool(async) vnodes = [] names = [names] if !names.is_a?(Array) names.each_index { |i| name = names[i] vnode = vnode_get(name) desc = descs.is_a?(Array) ? descs[i] : Marshal.load(Marshal.dump(descs)) downkeys(desc) vnode.sshkey = desc['ssh_key'] if desc['ssh_key'] and \ (desc['ssh_key'].is_a?(Hash) or desc['ssh_key'].nil?) vnode_attach(vnode.name,desc['host']) if desc['host'] if desc['vfilesystem'] vfs = desc['vfilesystem'] #image has to be unescaped since it was escaped by the coordinator vfs['image'] = CGI.unescape(vfs['image']) vfilesystem_create(vnode.name,vfs) end if desc['vcpu'] if vnode.vcpu vcpu_update(vnode.name,desc['vcpu']) else vcpu_create(vnode.name,desc['vcpu']) end end vnode.vmem = desc['vmem'] if desc['vmem'] if desc['vifaces'] desc['vifaces'].each do |ifdesc| if vnode.get_viface_by_name(ifdesc['name']) viface_update(vnode.name,ifdesc['name'],ifdesc) else viface_create(vnode.name,ifdesc['name'],ifdesc) end end end vnode_mode_update(vnode.name,desc['mode']) if desc['mode'] vnode_status_update(vnode.name,desc['status'],async) if desc['status'] vnodes << vnode } return vnodes end |
- (Object) vnodes_execute(names, command)
1068 1069 1070 1071 1072 1073 1074 1075 1076 |
# File 'lib/distem/daemon/distempnode.rb', line 1068 def vnodes_execute(names, command) w = Distem::Lib::Synchronization::SlidingWindow.new(100) names.each { |vnode_name| vnode = vnode_get(vnode_name) w.add("ssh -q -o StrictHostKeyChecking=no #{get_default_iface_from_vnode(vnode).address.address.to_s} \"#{command}\"", vnode_name) } w.run return w.results end |
- (Object) vnodes_freeze(names, async)
394 395 396 397 398 399 400 401 402 403 404 405 406 407 |
# File 'lib/distem/daemon/distempnode.rb', line 394 def vnodes_freeze(names, async) vnodes = [] names.each { |name| vnode = vnode_get(name) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) raise Lib::BusyResourceError, vnode.name if vnode.status == Resource::Status::CONFIGURING raise Lib::UninitializedResourceError, vnode.name if vnode.status == Resource::Status::INIT vnode.status = Resource::Status::CONFIGURING @node_config.vnode_freeze(vnode) vnode.status = Resource::Status::FROZEN vnodes << vnode } return vnodes end |
- (Object) vnodes_get
Get the list of the the currently created virtual nodes
Returns
Array of Resource::PNode objects
Exceptions
450 451 452 |
# File 'lib/distem/daemon/distempnode.rb', line 450 def vnodes_get() return @node_config.vplatform.vnodes end |
- (Object) vnodes_remove(names)
Remove the given virtual nodes, or every if names is nil
Returns
Array of Resource::PNode objects
Exceptions
291 292 293 294 295 296 297 298 299 300 301 302 |
# File 'lib/distem/daemon/distempnode.rb', line 291 def vnodes_remove(names) #some vnodes might not be initialized, in this case, they are dropped vnodes = names ? names.map { |name| vnode_get(name, false) }.compact : vnodes_get().values tids = [] vnodes.each { |vnode| tids << Thread.new { vnode_remove(vnode.name) } } tids.each { |tid| tid.join } return vnodes end |
- (Object) vnodes_stop(names, async = false)
214 215 216 217 218 219 220 221 222 223 224 225 |
# File 'lib/distem/daemon/distempnode.rb', line 214 def vnodes_stop(names,async=false) async = parse_bool(async) vnodes = names ? names.map { |name| vnode_get(name)} : vnodes_get().values tids = [] vnodes.each{ |vnode| tids << Thread.new { vnode_status_update(vnode.name, Resource::Status::DOWN) } } tids.each { |tid| tid.join } if !async return vnodes end |
- (Object) vnodes_unfreeze(names, async)
409 410 411 412 413 414 415 416 417 418 419 420 421 |
# File 'lib/distem/daemon/distempnode.rb', line 409 def vnodes_unfreeze(names, async) vnodes = [] names.each { |name| vnode = vnode_get(name) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) raise Lib::BusyResourceError, vnode.name if vnode.status != Resource::Status::FROZEN vnode.status = Resource::Status::CONFIGURING @node_config.vnode_unfreeze(vnode) vnode.status = Resource::Status::RUNNING vnodes << vnode } return vnodes end |
- (Object) voutput_get(vnodename, vifacename, raising = true)
654 655 656 657 658 659 660 |
# File 'lib/distem/daemon/distempnode.rb', line 654 def voutput_get(vnodename,vifacename, raising = true) viface = viface_get(vnodename,vifacename) raise Lib::UninitializedResourceError, 'voutput' if raising and !viface.voutput return viface.voutput end |
- (Object) voutput_update(vnodename, vifacename, desc)
662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 |
# File 'lib/distem/daemon/distempnode.rb', line 662 def voutput_update(vnodename,vifacename,desc) vnode = vnode_get(vnodename) raise Lib::ResourceError, "Please, contact the good PNode" unless target?(vnode) viface = viface_get(vnodename,vifacename) downkeys(desc) if desc and !desc.empty? viface.voutput = Resource::VIface::VTraffic.new(viface, Resource::VIface::VTraffic::Direction::OUTPUT,desc) else viface.voutput = nil end if vnode.status == Resource::Status::RUNNING vnode.status = Resource::Status::CONFIGURING @node_config.vnode_reconfigure(vnode) vnode.status = Resource::Status::RUNNING end return viface.voutput end |
- (Object) vroute_create(networksrc, networkdst, nodegw)
Create a virtual route (“go from <networkname> to <destnetwork> via <gatewaynode>”). The virtual route is applied to all the vnodes of <networkname>. This method automagically set <gatewaynode> in gateway mode (if it's not already the case) and find the right virtual interface to set the virtual route on
Attributes
-
destnetwork
the name of the destination network -
gatewaynode
the name of the virtual node to use as a gateway
Deprecated: * vnode
the virtual node to set the virtual route
on (optional)
Returns
Resource::VRoute object
Exceptions
960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 |
# File 'lib/distem/daemon/distempnode.rb', line 960 def vroute_create(networksrc,networkdst,nodegw) begin srcnet = vnetwork_get(networksrc) destnet = vnetwork_get(networkdst) begin gw = IPAddress.parse(nodegw) rescue ArgumentError raise Lib::InvalidParameterError, nodegw end gwaddr = gw #destnet = @node_config.vplatform.get_vnetwork_by_address(networkdst) unless destnet #destnet = vnetwork_create(nil,networkdst) unless destnet raise Lib::ResourceNotFoundError, networksrc unless srcnet raise Lib::ResourceNotFoundError, networkdst unless destnet raise Lib::ResourceNotFoundError, nodegw unless gw raise Lib::InvalidParameterError, nodegw unless gwaddr vroute = srcnet.get_vroute(destnet) unless vroute vroute = Resource::VRoute.new(srcnet,destnet,gwaddr) srcnet.add_vroute(vroute) end return vroute rescue Lib::AlreadyExistingResourceError raise rescue Exception destroy(vroute) if srcnet raise end end |
- (Object) vtraffic_update(vnodename, vifacename, desc)
Configure the virtual traffic on a virtual network interface, replacing previous values
Attributes
-
vnodename
The VNode name (String) -
vifacename
The VIface name (String) -
desc
Hash that represents the VTraffic description (see Lib::Validator)
Returns
Resource::VIface object
Exceptions
611 612 613 614 615 616 617 618 619 620 621 |
# File 'lib/distem/daemon/distempnode.rb', line 611 def vtraffic_update(vnodename,vifacename,desc) vnode = vnode_get(vnodename) viface = viface_get(vnodename,vifacename) downkeys(desc) voutput_update(vnode.name,viface.name,desc['output']) if desc['output'] vinput_update(vnode.name,viface.name,desc['input']) if desc['input'] return viface end |
- (Object) wait_vnodes(opts)
1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 |
# File 'lib/distem/daemon/distempnode.rb', line 1078 def wait_vnodes(opts) port = opts['port'] timeout = opts['timeout'] vnodes = [] if opts['vnodes'] vnodes = opts['vnodes'].collect { |vnode| get_default_iface_from_vnode(vnode_get(vnode)).address.address.to_s } else @node_config.vplatform.vnodes.each_value { |vnode| vnodes << get_default_iface_from_vnode(vnode).address.address.to_s } end finished = false begin Timeout::timeout(timeout) { while !finished tmp = [] vnodes.each { |ip| if !port_open?(ip,port) tmp << ip end } if !tmp.empty? vnodes = tmp sleep(10) else finished = true end end } rescue Timeout::Error return ['false'] end return ['true'] end |