学步园

Using an rsize


or wsize


larger than your network's MTU (often set to 1500, in many networks)
will cause IP packet fragmentation when using NFS over UDP. IP packet
fragmentation and reassembly require a significant amount of CPU
resource at both ends of a network connection. In addition, packet
fragmentation also exposes your network traffic to greater
unreliability, since a complete RPC request must be retransmitted if a
UDP packet fragment is dropped for any reason. Any increase of RPC
retransmissions, along with the possibility of increased timeouts, are
the single worst impediment to performance for NFS over UDP.

Packets may be dropped for many reasons. If your network topography
is complex, fragment routes may differ, and may not all arrive at the
Server for reassembly. NFS Server capacity may also be an issue, since
the kernel has a limit of how many fragments it can buffer before it
starts throwing away packets. With kernels that support the /proc

filesystem, you can monitor the files /proc/sys/net/ipv4/ipfrag_high_thresh

and /proc/sys/net/ipv4/ipfrag_low_thresh

. Once the number of unprocessed, fragmented packets reaches the number specified by ipfrag_high_thresh

(in bytes), the kernel will simply start throwing away fragmented
packets until the number of incomplete packets reaches the number
specified by ipfrag_low_thresh

.

Another counter to monitor is IP: ReasmFails

in the file /proc/net/snmp

;
this is the number of fragment reassembly failures. if it goes up too
quickly during heavy file activity, you may have a problem.

A new feature, available for both 2.4 and 2.5 kernels but not yet
integrated into the mainstream kernel at the time of this writing, is
NFS over TCP. Using TCP has a distinct advantage and a distinct
disadvantage over UDP. The advantage is that it works far better than
UDP on lossy networks. When using TCP, a single dropped packet can be
retransmitted, without the retransmission of the entire RPC request,
resulting in better performance on lossy networks. In addition, TCP will
handle network speed differences better than UDP, due to the underlying
flow control at the network level.

The disadvantage of using TCP is that it is not a stateless protocol
like UDP. If your server crashes in the middle of a packet transmission,
the client will hang and any shares will need to be unmounted and
remounted.

The overhead incurred by the TCP protocol will result in somewhat
slower performance than UDP under ideal network conditions, but the cost
is not severe, and is often not noticable without careful measurement.
If you are using gigabit ethernet from end to end, you might also
investigate the usage of jumbo frames, since the high speed network may
allow the larger frame sizes without encountering increased collision
rates, particularly if you have set the network to full duplex.

Two mount command options, timeo

and retrans

,
control the behavior of UDP requests when encountering client timeouts
due to dropped packets, network congestion, and so forth. The -o timeo

option allows designation of the length of time, in tenths of seconds,
that the client will wait until it decides it will not get a reply from
the server, and must try to send the request again. The default value is
7 tenths of a second. The -o retrans

option allows designation of the number of timeouts allowed before the
client gives up, and displays the Server not responding message. The
default value is 3 attempts. Once the client displays this message, it
will continue to try to send the request, but only once before
displaying the error message if another timeout occurs. When the client
reestablishes contact, it will fall back to using the correct retrans

value, and will display the Server OK

message.

If you are already encountering excessive retransmissions (see the output of the nfsstat

command), or want to increase the block transfer size without
encountering timeouts and retransmissions, you may want to adjust these
values. The specific adjustment will depend upon your environment, and
in most cases, the current defaults are appropriate.

Most startup scripts, Linux and otherwise, start 8 instances of nfsd

.
In the early days of NFS, Sun decided on this number as a rule of
thumb, and everyone else copied. There are no good measures of how many
instances are optimal, but a more heavily-trafficked server may require
more. You should use at the very least one daemon per processor, but
four to eight per processor may be a better rule of thumb. If you are
using a 2.4 or higher kernel and you want to see how heavily each nfsd

thread is being used, you can look at the file /proc/net/rpc/nfsd

. The last ten numbers on the th

line in that file indicate the number of seconds that the thread usage
was at that percentage of the maximum allowable. If you have a large
number in the top three deciles, you may wish to increase the number of nfsd

instances. This is done upon starting nfsd

using the number of instances as the command line option, and is specified in the NFS startup script (/etc/rc.d/init.d/nfs

on Red Hat) as RPCNFSDCOUNT

. See the nfsd(8)

man page for more information.

On 2.2 and 2.4 kernels, the socket input queue, where requests sit
while they are currently being processed, has a small default size limit
(rmem_default

) of 64k. This
queue is important for clients with heavy read loads, and servers with
heavy write loads. As an example, if you are running 8 instances of nfsd
on the server, each will only have 8k to store write requests while it
processes them. In addition, the socket output queue - important for
clients with heavy write loads and servers with heavy read loads - also
has a small default size (wmem_default

).

Several published runs of the NFS benchmark SPECsfs97
specify usage of a much higher value for both the read and write value sets, [rw]mem_default and [rw]mem_max

.
You might consider increasing these values to at least 256k. The read
and write limits are set in the proc file system using (for example) the
files /proc/sys/net/core/rmem_default

and /proc/sys/net/core/rmem_max

. The rmem_default

value can be increased in three steps; the following method is a bit of
a hack but should work and should not cause any problems:

If network cards auto-negotiate badly with hubs and switches, and
ports run at different speeds, or with different duplex configurations,
performance will be severely impacted due to excessive collisions,
dropped packets, etc. If you see excessive numbers of dropped packets in
the nfsstat

output, or
poor network performance in general, try playing around with the network
speed and duplex settings. If possible, concentrate on establishing a
100BaseT full duplex subnet; the virtual elimination of collisions in
full duplex will remove the most severe performance inhibitor for NFS
over UDP. Be careful when turning off autonegotiation on a card: The hub
or switch that the card is attached to will then resort to other
mechanisms (such as parallel detection) to determine the duplex
settings, and some cards default to half duplex because it is more
likely to be supported by an old hub. The best solution, if the driver
supports it, is to force the card to negotiate 100BaseT full duplex.

The default export behavior for both NFS Version 2 and Version 3 protocols, used by exportfs

in nfs-utils versions prior to nfs-utils-1.0.1 is "asynchronous". This
default permits the server to reply to client requests as soon as it has
processed the request and handed it off to the local file system,
without waiting for the data to be written to stable storage. This is
indicated by the async

option denoted in the server's export list. It yields better performance
at the cost of possible data corruption if the server reboots while
still holding unwritten data and/or metadata in its caches. This
possible data corruption is not detectable at the time of occurrence,
since the async

option
instructs the server to lie to the client, telling the client that all
data has indeed been written to the stable storage, regardless of the
protocol used.

In order to conform with "synchronous" behavior, used as the default
for most proprietary systems supporting NFS (Solaris, HP-UX, RS/6000,
etc.), and now used as the default in the latest version of exportfs

,
the Linux Server's file system must be exported with the sync option.
Note that specifying synchronous exports will result in no option being
seen in the server's export list:

If your kernel is compiled with the /proc

filesystem, then the file /proc/fs/nfs/exports

will also show the full list of export options.

When synchronous behavior is specified, the server will not complete
(that is, reply to the client) an NFS version 2 protocol request until
the local file system has written all data/metadata to the disk. The
server will complete a synchronous NFS version 3 request without this
delay, and will return the status of the data in order to inform the
client as to what data should be maintained in its caches, and what data
is safe to discard. There are 3 possible status values, defined an
enumerated type, nfs3_stable_how

, in include/linux/nfs.h

. The values, along with the subsequent actions taken due to these results, are as follows:

In addition to the above definition of synchronous behavior, the
client may explicitly insist on total synchronous behavior, regardless
of the protocol, by opening all files with the O_SYNC

option. In this case, all replies to client requests will wait until
the data has hit the server's disk, regardless of the protocol used
(meaning that, in NFS version 3, all requests will be NFS_FILE_SYNC

requests, and will require that the Server returns this status). In
that case, the performance of NFS Version 2 and NFS Version 3 will be
virtually identical.

If, however, the old default async

behavior is used, the O_SYNC

option has no effect at all in either version of NFS, since the server
will reply to the client without waiting for the write to complete. In
that case the performance differences between versions will also
disappear.

Finally, note that, for NFS version 3 protocol requests, a subsequent
commit request from the NFS client at file close time, or at fsync()

time, will force the server to write any previously unwritten
data/metadata to the disk, and the server will not reply to the client
until this has been completed, as long as sync behavior is followed. If async

is used, the commit is essentially a no-op, since the server once again
lies to the client, telling the client that the data has been sent to
stable storage. This again exposes the client and server to data
corruption, since cached data may be discarded on the client due to its
belief that the server now has the data maintained in stable storage.

In general, server performance and server disk access speed will have
an important effect on NFS performance. Offering general guidelines for
setting up a well-functioning file server is outside the scope of this
document, but a few hints may be worth mentioning: