TIPC Debugging Guide – Network Troubleshooting
The Transparent Inter-Process Communication protocol (TIPC) allows applications in a clustered computer environment to communicate quickly and reliably with other applications, regardless of their location within the cluster. TIPC originated at the telecommunications manufacturer, Ericsson, and has been deployed in their products for years.
This document is intended to assist developers in debugging and troubleshooting
TIPC problems.
Debugging problems with TIPC or with applications using TIPC typically involves
the following sequence of steps:
- Recognizing the existence of the problem
- Gathering information about what is happening at the time of the problem
- Determining the cause of the problem and
- Developing a solution.
This document focuses on capabilities provided by TIPC for dealing with the
first three steps (i.e. the information gathering stage).
1. tipc-config
The tipc-config tool provides a number of options that allow a user to query
TIPC in order to determine its current state. Displaying information about
the state of a node’s links, ports, and name table can often be used to
identify whether or not portions of TIPC are working as expected. For example:
- The “-l” option allows you to see the state of TIPC links. A link that is
down (or missing) may indicate a problem.- The “-ls” option allows you to see detailed information about a specific
TIPC link, which may be helpful in identifying cases where links are not
functioning smoothly.- The “-nt” option allows you to see the contents of TIPC’s name table. Ensure
that the entries your applications have published are present, and that no
unexpected entries are present.CAUTION: Remember that TIPC may publish names using the reserved types
0 through 63. Each node publishes an entry of the form {0, x} for its
configuration service (where “x” is the decimal representation of a node’s
network address,), and every node’s name table should contain entries
for all active nodes in the cluster. There should also be exactly one entry
of the form {1, 1} which is published by TIPC’s topology service.
For a more complete description of the tipc-config tool and its options, see
the TIPC User’s Guide.
2. Socket Return Codes
TIPC’s socket API is designed to return error values in accordance with the
POSIX specification. If your application detects that a socket-related routine
has unexpectedly failed it is a good to print out the “errno” value, as this
may provide useful information about the cause of the problem.
3. TIPC System Messages
TIPC automatically generates a “system message” whenever it detects the
occurance of a significant event. Three categories of events are recorded:
- informatory: a normal condition that TIPC has successfully handled eg. a link comes up or goes down
- warning: an abnormal condition that TIPC was able to recover from eg. invalid user request
- error: an abnormal condition that TIPC may be unable to fully recover from eg. corruption of an internal TIPC data structure
TIPC’s system messages can be used both to identify the existence of new
problems and to debug known problems, and should be checked on a regular basis.
Unless otherwise changed by the TIPC system administrator, TIPC’s system
messages are sent to both the operating system’s console and TIPC’s own log
buffer (see below).
3.1 The System Console
The system console is an operating system-dependent mechanism for displaying
information from many parts of the OS, including TIPC.
On many Linux systems, messages that are sent to the system console are saved
in /var/log/messages.
3.2 The TIPC Log Buffer
The TIPC log buffer is a circular buffer that stores only output from TIPC.
To display the log, use “tipc-config -log”. CAUTION: This is a destructive
process — the contents of the log buffer are erased after it has displayed.
By default the log buffer has a size of 0, and will record no system messages.
To increase the buffer size, use “tipc-config -log=N”, where N is the number of
bytes to allocate (up to 32768). CAUTION: This is a destructive process –
the contents of the log buffer are erased when the size is changed.
NOTE: Unlike the messages in Linux’s system console file, TIPC system messages
in the log buffer do not currently have any timestamp indicating when they
were issued.
3.3 Adding New System Messages
In certain instances, users may wish to modify or augment TIPC’s system
messages. This requires the user to modify the TIPC source code, then rebuild
and reinstall TIPC on affected nodes.
TIPC uses 3 macros to generate system messages:
info(fmt, arg...); /* print informatory message */
warn(fmt, arg...); /* print warning message */
err(fmt, arg...); /* print error message */
Each of these macros takes that same arguments as printf(), so it is easy
to display information in an easily readable manner.
4. TIPC Debug Messages
In addition to its system messages, TIPC can also generate additional debugging
messages. However, this requires the user to modify the TIPC source code,
then rebuild and reinstall TIPC on affected nodes.
4.1 Enabling Existing Debugging Messages
TIPC provides 2 macros to generate debugging messages:
dbg(fmt, arg...); /* print debug message */
msg_dbg(msg, txt); /* display TIPC message header info */
The first macro takes the same arguments as printf(). The second argument
takes a pointer to the TIPC message header (“msg”) and a pointer to an ASCII
string (“txt”) that is printed out along with the messsage header info.
These macros send their output to the TIPC print buffer indicated by the
symbol DBG_OUTPUT. By default this symbol is set to NULL, which prevents
debug messages from being generated; to enable debug output it is necessary
to redefine this symbol for the places where debugging code is to be enabled.
For example, to send selected debugging messages to the system console:
... [area for which debugging messages ARE NOT to be printed]
#undef DBG_OUTPUT
#define DBG_OUTPUT CONS
... [area for which debugging messages ARE to be printed]
#undef DBG_OUTPUT
#define DBG_OUTPUT NULL
... [area for which debugging messages ARE NOT to be printed]
Debug messages can be sent to the TIPC log buffer by replacing CONS with LOG
in the example above. To send output to both locations, use TEE(CONS,LOG).
CAUTION: To avoid generating a vast flood of debugging messages, it is usually
a good idea to enable debugging message only for the routine (or routines)
you are interested in observing, rather than enabling debugging for an entire
file.
4.2 Adding New Debugging Messages
In certain instances, users may wish to enhance TIPC’s debugging messages.
This can be done by simply introducing new code that calls the dbg() and
msg_dbg() macros mentioned in the previous section.
Alternatively, in cases where a user simply wants to add a simple printf()-like
statement that will be thrown away after the problem is fixed, one of the
following two routines can be used instead:
printk(fmt, arg...); /* print message to system console */
tipc_printf(LOG, fmt, arg...); /* print message to TIPC log buffer */
The advantage of these two forms is that it is not necessary to redefine
DBG_OUTPUT to enable or disable output; however, the only way to disable output
for these calls is to remove them (or use conditional compilation to hide them).
Original Source : http://tipc.sourceforge.net/doc/Debug_Guide.txt
Related Posts:
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.


Leave a comment.