Comparision of Network-Simulators
The Small PC-Network model consists of two routers connected via a bottleneck link, which can transmit 20 packets per second. Each routers serves several hosts (from 18 to 22) which are Poisson sources with a rate of 1 packet per second. So the inter-arrival time of the sent packets are exponentially distributed.
Both routers have a limited queue size, which is configurable and part of the evaluation. The queues are simple FIFO queues, where packets received while the queue is full, are simply dropped. The packet loss should be evaluated over various queue sizes and various host numbers (18, 20, 22). All packets are assumed to have the same size, so that service time is also constant.
The simulation tools:
OMNet++ is a public-source simulation environment, which main goal is the simulation of communication networks. However its design is quite open, which enables also other target applications. It has a sophisticated GUI support and common used models like IPv4, IPv6, Ethernet, MPLS etc. are available. OMNet++ is free for academic and non-profit use, however for commercial use a license must be obtained.
OMNet++ modules are structured by an own network definition language NED, while the functionality is coded by using C++ classes. OMNet++ is available for Unix-Systems (like Linux) as well as for Windows supporting Visual C++ 6.0, Visual Studio .NET 2003 and CygWin.
The network simulator 2 (ns-2) is a frequently used discrete event simulator mainly for networking research. For implementing the model, the object orientated extension of TCL, OTcl is used. The functionality of modules can also be coded in OTcl or for speed optimizations in C++. ns-2 includes the most common network technologies and applications, for very easy and fast network specification and simulations.
ns-2 is also free available for research and education issues. It is designed for Unix-Systems but runs under Windows CygWin as well.
QualNet is the first commercial simulator in this comparison. It is based on GloMoSim developed at the University of California, Los Angeles (UCLA). GloMoSim uses the Parallel Simulation Enviroment for Complex Systems (PARSEC) for basic operations.
QualNet also has a graphical user interface for creating the model and its specification. So it is by far easier to specify small to medium networks by using the GUI compared to specifying all connections in a special model file manually.
QualNet is a network simulator targeting at wireless solutions, however it also has support for wired networks. Its environment and library is very sophisticated, which makes it very easy to simulate a real network with QualNet. This however makes the simulation of logical networks a little bit more difficulty, but it is possible as well.
Since it uses primarily Java for the GUI it is available for Linux as well as for Windows. The simulator itself is a for the specified target system optimized C program.
SimPy is an extension for the script language Python for easy discrete event simulation. The advantage is a easy modifiable language for discrete event simulation without any restrictions, however the simulation performance is lower, since Python is still a script language, which is interpreted.
It is released under the GNU Library or LGPL license, which enables everyone free use of the package. Since its an extension for Python, its available on every system, where Python is running which includes Linux and Windows.
Enterprise Dynamics V6.1 Build 497
Entprise Dynamics is a simulator for industrial processes. It has a very nice GUI for developing the model as well as simulating it afterwards. The core modules can be extended by using the own scripting language 4DScript. Since its main target is not network simulation, its performance is not well optimized for this task, but it is still possible.
It is also a commercial product like QualNet, but it is only available for Windows.
AnyLogic is another commercial product. However it is the only simulator here, which supports discrete, continous and hybrid simulation methods. However only the discrete part was used for this evaluation. The GUI is very sophisticated enabling fast model design as well as interactive experimentation.
AnyLogic is based on Java. So every module has the possibilty to be enhanced by custom Java functions. However the simulator is only available for Windows.
Implementation:The implementation were quite different for each simulator, however the results were always the same, like it could be observed in the next section.
The model in OMNet++ was designed modular. So a router module was created consisting of a FIFO-queue with n input values and one output, and a sink for incoming messages over the bottleneck. Then a network was created consisting of 18-22 hosts and one router, and two instances of this network was connected over the bottleneck link.
The modules for the FIFO-queue, the random traffic generator and the sink were derived from modules of the OMNet++ examples. The FIFO-queue was made finite with counters for incoming and dropped packets. The sending functions of the FIFO-queue was also modified, to send only, if the bottleneck link is not currently busy.
Overall this were small and easy changes, however it needed quite some time to work into the system of the simulator, to understand all parts of it. Overall it maybe took the longest to develop model for OMNet++ but the comfort of the TK GUI then was one of the best.Source
The implementation was taken from . This implementation did not result in the right values, since the implementation of a ns-2 queue is somewhat weird. Effectively ns-2 queues with one element can never hold any item, and will always produce 100% packet loss, which is not the correct behavior. A look into the ns-2 sources revealed, that the queue size in ns-2 always need to be one bigger than usual, to result in the same behavior.
The hosts were simulated by using UDP-agents directly connected to the router. This agent produces Poisson traffic by using the Traffic/Expoon module. The queue monitors collect the statistic of the two router queues, and about received and dropped packets. With this values the packet loss will be computed after finishing the simulation.Source
Here in QualNet only one side of the Small PC-Network was simulated. Actually 22 hosts were modeled, with two routers. All 22 hosts are connected to the first router, and between the two routers there is the bottleneck link. Depending on the number of hosts to participate, 18, 20 or 22 traffic generators are used as applications, which sends fixed sized packets from the hosts to the second router. The link between the hosts and the first router where modeled as 100 MBit/s links, while the bottleneck link had a bandwidth of 80040 Bit/s. The additional 40 Bit/s were needed since QualNet requires a link level header to be used (with greater than 0 bits). A two bit link level header was used over the bottleneck link. So for 20 Packets/s this results in additional 40 Bit/s.
The traffic generator application did generate packets with 472 bytes of data size plus 28 bytes for the IP- + UDP-headers. This resulted in packets with 500 Bytes. The router queue in the first router was just set to a multiple of 500 Bytes.Source
The previous simulators were all dedicated for network simulation. SimPy however is a general process-based discrete event simulator. So two different kind of processes were created. One process called "Host" just creates messages with exponential inter-departure times. This process then calls a procedure of the second process "Router" to transmit the message. The "Router" process manages the queue, and takes packets with a rate of 20 packets per second out of the router queue. The "Router" process keeps track of received and dropped packets, and so can supply a packet loss ratio at the end of the simulation.Source
Like mentioned above Enterprise Dynamics is a simulator for industrial processes. Therefore the objects used for modeling the small PC-Network were a little bit different.
Product sources were used as hosts, which created products with exponential inter-departure times. Then a special queue was used, which sends a packet prioritized to the first link, which is our router queue. If this link is occupied (means if the router queue is full) the packets are sent to the second link, which is the lost packets sink. Behind the queue a link with 20 packets per second was connected, which leads to another sink.Source
Results:Due to the different simulation speeds of the simulators, we did not simulate the same number of seconds with every simulator. In the upper right legend on every plot there is the number of simulated seconds inside the brackets (10^6 means 1000000, 10^5 = 100000 and 10^4 = 10000).
As you can see, the results of OMNet++, ns-2 and QualNet are so close together, that there cant be observed any differences in the result plots. The SimPy results vary a little bit, since only a tenth of time of the other simulators were simulated. (Last Enterprise Dynamics only had very few point because of the slow simulation and this also with much less simulation seconds. So the result of this is the bigger variations, but still the model behaves in the same way, as the other ones does.)
Since the simulation results are so close together, it is rather important which simulator needed how much time for getting the this results.
The performance was measured on the same PC (a Pentium4 2.4GHz without Hyper-Threading with 1GB RAM). The results shown below are for the simulation run of 22 hosts and a queue size of 200 packets. Due to modeling and speed issues not all simulation models were designed with both sides. So there were models which were only one-sided.
Enterprise Dynamics and AnyLogic were evaluated with the Windows XP Professional SP 2 operating system. All other simulators used a Fedora Core 3 installation with Linux 2.6.12-1. For SimPy the evaluation was done with and without the usage of psyco. Psyco is an extension of Python to speed up its execution.
|Simulator||simulated seconds||simulated sides||runtime|
|SimPy (with psyco)||100000||1||3min 2s|
|SimPy (without psyco)||100000||1||4min 8s|
|Enterprise Dynamics||10000||1||2min 15s|
Since the two-sided models generate approx. double of the events, the runtime of a one-sided model is considered to be the half of the measures two-sided model. So this results in following performance values:
|Simulator||simulation seconds per real-time second||simulation events per real-time second|
|SimPy (with psyco)||578.03||n/a|
|SimPy (without psyco)||403.23||n/a|
This shows by far OMNet++ as the winner of this performance comparison. However the effort needed to create the simulation in OMNet++ is quiet bigger than in all other simulators, but OMNet++ provides the possibility to create the simulation from the scratch with good speed optimizations, while a comprehensive INET package is available as well.
The commercial simulators QualNet and Enterprise Dynamics both supplies a graphical design mode, which shows that QualNet is quiet usable for network simulation beginners as well, but it still provides many pitfalls. Since QualNet's main purpose are wireless simulations, you need to be careful which settings are enabled by default. It is also a tool which targets practitioners. So it supplies many advanced and comprehensive features, which are not needed in logical simulation, and which are even hard to get rid of.
Enterprise Dynamics is targeted for simulation of industrial processes, and therefore does not provide such a good performance. With its good sophisticated graphical user-interface it was by far the easiest tool for modeling the problem. The performance from Enterprise Dynamics was somewhat disappointing. Another performance run for only 18 hosts with queue size 200 only lasted 3min 17s. So Enterprise Dynamics does not seem to scale very well or has some functions which were more often used for higher utilization, which are very inefficient.
Last SimPy is only a discrete event simulation extension for Python, which can be used for any discrete simulations. It is quiet easy to program since Python also provides sophisticated methods and data structures. However it is better suited for smaller projects or just for small and fast experimentation of simulation models.
My personal favorite is OMNet++, since it is has the most clear and open programming interface (while ns-2 has some restrictions in some parts), and also supplies with the compiled in graphical user-interface (with TK), where packet flow can be observed and data can be modified during a simulation run. So the experimentation features are also better than ns-2. However changing the model is more work than in ns-2, since ns-2 provides the OTcl script interface, where the model and also some modules can be coded quiet fast.
The vocational adjustment is lowest for the two commercial products with integrated GUI designer mode.