Next: Shared Networks
Up: Parallel Programs
OK, by now you should be getting the hang of things. A beowulf is a
parallel supercomputer built out of COTS nodes interconnected by a COTS
network of some kind. One can build a beowulf to speed up a piece of
parallelized code (in the classic Amdahlian sense) so it finishes
faster. One can build a beowulf to be able to do a task at all by
assembling more resources than one can either afford any other way or
than are currently available in a system at any price. One can build a
beowulf to speed up a code in an exotic way (by providing a faster
extended virtual memory space, for example).
In the previous chapter we discussed all sorts of ways the basic
bottlenecks between the CPU and memory subsystems (within a node, by
assumption) can affect program speed, trying to provide a
semi-quantitative understanding so that you can at least do the back of
the envelope calculations required to compare the cost-benefit of
various alternative ways of accomplishing a task. In this chapter we'll
focus on the sine qua non bottleneck of beowulfery, the network.
There is so much to learn about networking and how it relates to
serious beowulfery that it is hard to know just how much to put into an
introductory book like this. To invert the point, there is such a wide
range of ignorance about networking out there that I could easily be
speaking to someone who doesn't know Appletalk from Ethernet, has never
heard of the ISO or OSI, for whom TCP and IP are a mystery, and who
thinks that a router is a device for cutting interesting curves in a
piece of wood.
If this is you: Sorry, chum, you won't learn about these things here,
or at least you won't learn much (certainly not enough to assemble a
functional linux network). What can I say - there are whole books that
focus on just setting up and running a network, and I cannot
compress all that into a chapter and have time to say anything at all
about networking in the fairly strict context of beowulfery.
So, even though a network is key to a beowulf, I'm going to assume that
in fact you do know what the following are:
- NIC (Network Interface Card), typically a PCI (regular, fast, or
gigabit) ethernet adaptor or PCI Myrinet adaptor, although there are now
some exotic alternatives with more on the horizon. This little pup
plugs into the PCI bus of your node and is connected via RJ45 cables to
- Hub. This is something that distributes a signal in the transmit
wires of one connection to the receive wires of all the other
connections. There are all sorts of technical things associated with
hubs (like the number of hubs you can put in between hosts). In
general, hubs are ``bad'' in beowulf design unless one's needs are
modest and likely to stay that way. If one can afford it, one would
usually do better with a
- Switch. This is something that establishes a virtual pairwise
connection between hosts plugged into the switch. Where the hub allows
only one host at a time to talk in one direction, a switch
allows all the hosts to talk at the same time in both
directions. In principle - be sure to check the ``bisection
bandwidth'' of your switch to ensure that it can indeed handle
full-duplex on all lines at once as a cheap switch might not.
- Ethernet. The most common (and cheapest) networking hardware
protocol. Ethernet comes with an amazing and complex set of rules and
standards, such as how long a cable can be used to connect a host to a
hub or switch, how many hubs can exist between hosts, what an adaptor
has to do in the event of a collision (what a collision is in the first
place), in addition to certain baseline latencies and bandwidths and
packet sizes. Can't know too much about ethernet, no sirree-bob!
- Myrinet. The most common and probably premier gigabit network.
It's about ten times faster than 100BT ethernet, and costs about twenty
times as much per node (or even a bit more).
- IP. ``Internet Protocol''. Yes, it is the fundamental protocol
upon which the internet is founded. Associated with IP is the
``internet address'' (or IP address) of a host (adaptor), a packet
structure (headers and so forth), and above all a routing and filtering
- TCP. ``Transmission Control Protocol''. This lives ``on top of''
IP and regulates things like the reliable delivery of packets across an
uncertain network. Most common services (like mail, ftp, telnet, http
and so forth) are defined in terms of standard port addresses within
TCP. TCP is a bit smarter and more reliable than ``raw'' IP, and hence
is a tiny bit slower.
- Sockets. A ``socket'' is an abstraction for a network connection.
One opens up a socket on a host and a remote service connects to that
socket. Information flows between the sockets. Sockets can be read or
written to like files (of course, in Unix and hence linux, everything is
a file, right?).
- UDP. ``Unix Datagram Protocol''. This is a kind of socket and
hence network connection. It is the ``raw IP'' connection I referred to
earlier. A TCP socket is more reliable but has certain limitations and
costs. UDP is commonly used to provide local services within a local
area network LAN), TCP across a wide area network or WAN. Except that
isn't necessarily true anymore, as reliability is almost always more
important than raw speed. NFS is probably the most famous UDP service.
- Router. Something that routes packets, usually IP packets
(actually a router can often route lots of kinds of packets but we don't
care about any other kinds).
- Gateway. A router that typically lives between an ``inner''
network (LAN) and an ``outer'' network (ultimately the rest of the
Internet). It lets in the good packets and lets out the bad packets. A
true beowulf typically has a gateway node that is also usually a
``head'' node from which it is controlled and may also be a server node
if the internal nodes require (e.g.) NFS services. The gateway node can
keep spurious external traffic off the private internal network of the
As you can see, I am omitting all sorts of useful and important things.
You won't learn about netmasks, broadcasts, how to configure a NIC, or
any of that from me. However, I will direct you to the /usr/doc/HOWTO
directory (in most linux distributions) that has explicit step by step
instructions for setting up all sorts of things including the network.
Don't forget about Linux Headquarters (http://www.linuxhq.com/) either,
which has links to all the HOWTOs and other documentation. There are a
bunch of key learning documents in my own personal website including
http://www.phy.duke.edu/rgb/security/local.guide, which is ``the''
classic 1988 Rutgers white paper by Charles Hedrick describing all sorts
of networking concepts. Finally, there are a whole bunch of useful
URL's on the Brahma website (http://www.phy.duke.edu/brahma) which might
be of interest to the neophyte.
SO, from here on I'm going to assume that you can design and set up a
simple ethernet-based IP subnet without having your hand held. We'll
still address some of this sort of thing in the next chapter, but for
now we'll focus on the technical details (especially things like
latency, effect of packet size on bandwidth, problems, solutions) and
not on truly introductory things.
Next: Shared Networks
Up: Parallel Programs
Robert G. Brown