Glance reports "na" for logical I/Os

HP-UX

For the latest version of Glance (and MeasureWare/OVPA) logical I/Os are not being measured. In Glance, the logl Rds and loglWts are marked: "na". For MWA (MeasureWare or OVPA) there are no readings stored for logical (that is, memory resident) reads and writes. The midaemon process retrieves raw information from the kernel as event traces and these are gathered as quickly as possible by the ‘reader’ portion of the code. In order to buffer massive bursts of data, this raw information is stored in a shared memory area so the ‘writer’ portion of the code can independently write the data to the measurement interface. For large scale systems (16 or more processors, 4 Gbit or higher storage connections), the logical read/write rate can easily reach into the 100′s of thousands per second and thus overrun the midaemon memory area (too many reads, not enough writes). The error seen in the /var/opt/perf/status.scope file is:

GBL_LOST_MI_TRACE_BUFFERS

To fix this in C.04 versions, memory areas for midaemon must be adjusted.

CPUs     midaemon parameters
8-15    -bufsets 16 -skipbuf 8 -smdvss 256M
16-31   -bufsets 24 -skipbuf 12 -smdvss 256M
32-63   -bufsets 32 -skipbuf 16 -smdvss 512M
64+     -bufsets 32 -skipbuf 16 -smdvss 512M

Make sure your kernel parameter maxdsiz is at least 2048 MB or larger to accommodate the larger midaemon process. Additionally, the option -no_fileio_traces may have to be set for midaemon. This is done in the file /etc/rc.config.d/ovpa by adding (or modifying) the MIPARMS variable:

export MIPARMS = "-p -no_fileio_traces -bufsets 32 -skipbuf 16 -smdvss 512M"

The -p option makes midaemon permanent so the command line settings will not be lost when other performance tools are restarted. The -no_fileio_traces option will disable logical I/O metrics. Note that some installations may not have an ovpa startup script but have an mwa script.

To enable the new settings, use the commands:

ovpa stop
midaemon -T
ovpa start

For some installations, ovpa may not exist in which case you would edit the mwa file rather than the ovpa file. The setting would be:

MWA_START=1
# see the script /opt/perf/bin/mwa for special handling of midaemon parms
MWA_START_COMMAND="/opt/perf/bin/mwa start -midaemon -fileio_traces -p"

Verify that midaemon is running with the options: -fileio_traces -p and if not, manually run midaemon with the options. Starting with version C.05, the option -no_fileio_traces is the default value, resulting in "na" for Glance and no logical I/O metrics for MWA. However, midaemon was enhanced for C.05 and can handle a larger number of event traces so turning the option back on is a viable option. Just check the /var/opt/perf/status.scope
logfile for errors. The command line parameters for the ovpa startup config file would be:

export MIPARMS="-p -no_fileio_traces"

| Comments Off

Files in /var/spool/sockets/ICE?

HP-UX

Q: Is it OK to remove the files from /var/spool/sockets/ICE?

A: Yes, these files are leftover from running Xwindows (actually, running CDE). They are leftover when users do not logout and close their desktop (CDE) sessions properly.

If someone is running CDE, there will be a socket with the process ID number (PID). Those sockets should be left in place. To save time, here is a simple script to remove the unused files:

 #!/usr/bin/sh

 cd /var/spool/sockets/ICE

 /usr/bin/ls | while read X

 do

     kill -0 $X > /dev/null

     RTN=$?

     [[ $RTN -ne 0 ]] && rm $X

 done

Just put this into a file in /tmp and run it like this:

 sh /tmp/ICEcleanup

As a note to the above:  kill -0 (that’s zero, not the letter o)  simply verifies that the PID ($X in the above code) exists.  If not, then $RTN will be non-zero.

This avoids removing active sockets.

| Comments Off

HP-UX email with attachments

HP-UX

1. Mail No attachment, just the body:

mailx -m -s "My Subject" billh@abc.com \

        < file_for_email_body
					

2. Mail with attachement, and no body:

                                                                       
ux2dos /path_to/some_file |

   uuencode some_file.txt |

   mailx -m -s "MAIL FROM Billh" billh@abc.com

3. Mail with both attachment and body:

ux2dos /path_to/some_file |

   uuencode some_file.txt |

   mailx -m -s "My subject" billh@abc.com \

     < file_for_body

4. Mail with carbon (cc) and blind carbon (bcc):

mailx -m -s "Some Subject" billh@abc.com \

      < file_for_body

  (Note that this does not work well when using the -r option in mailx.)

  The file_for_email_body should look like this:

    ~c somebody_to_cc@abc.com

    ~b somebody_to_bc@abc.com

    The remainder of this file is the

    body of the email.

5. Mail with multiple attachments and body: 
   (Leave out the 'Here is some body text' if no body text is needed)

mailx -m -s "Hello" billh@abc.com << EOF                    

    $(ux2dos /home/blh/.kshrc |

    uuencode /home/blh/.kshrc.txt)   

    $(ux2dos /home/blh/.profile |

    uuencode /home/billh/.profile.txt)

Here is some body text. 

EOF                                              

| Comments Off

lvremove fails to remove lvol

HP-UX

Scenario:
A mismatch exists between the Cur LV and Open LV:

— Volume groups —
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 8
Open LV 9
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4384
VGDA 4
PE Size (Mbytes) 32
Total PE 8748
Alloc PE 1432
Free PE 7316
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

Looking at vgdisplay -v vg00, lvol9 is zero length.

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 0
Current LE 0
Allocated PE 0
Used PV 0

This is a valid size for LVM and normally this lvol can be removed, but:

# lvremove /dev/vg00/lvol9
lvremove: Couldn’t delete logical volume "/dev/vg00/lvol9″:
The supplied lv number refers to a non-existent logical volume.

But the device files look OK:

# ll
total 32
drwxr-xr-x 2 root root 8192 Apr 10 11:42 .
dr-xr-xr-x 16 bin bin 8192 Feb 24 20:31 ..
crw-r—– 1 root sys 64 0×000000 Jun 30 2011 group
brw-r—– 1 root sys 64 0×000001 Feb 24 20:30 lvol1
brw-r—– 1 root sys 64 0×000002 Jun 30 2011 lvol2
brw-r—– 1 root sys 64 0×000003 Jun 30 2011 lvol3
brw-r—– 1 root root 64 0×000004 Jul 5 2011 lvol4
brw-r—– 1 root sys 64 0×000005 Jun 30 2011 lvol5
brw-r—– 1 root sys 64 0×000006 Jun 30 2011 lvol6
brw-r—– 1 root sys 64 0×000007 Jun 30 2011 lvol7
brw-r—– 1 root sys 64 0×000008 Jun 30 2011 lvol8
brw-r–r– 1 root sys 64 0×000009 Apr 10 11:42 lvol9
crw-r—– 1 root sys 64 0×000001 Jun 30 2011 rlvol1
crw-r—– 1 root sys 64 0×000002 Jun 30 2011 rlvol2
crw-r—– 1 root sys 64 0×000003 Jun 30 2011 rlvol3
crw-r—– 1 root root 64 0×000004 Jul 5 2011 rlvol4
crw-r—– 1 root sys 64 0×000005 Jun 30 2011 rlvol5
crw-r—– 1 root sys 64 0×000006 Jun 30 2011 rlvol6
crw-r—– 1 root sys 64 0×000007 Jun 30 2011 rlvol7
crw-r—– 1 root sys 64 0×000008 Jun 30 2011 rlvol8
crw-r–r– 1 root sys 64 0×000009 Apr 10 11:41 rlvol9

So the LVM tables for lvol mapping are corrupted. There are a few circumstances where this can happen but it is easy to fix. Use rmsf (*NOT* rm) to remove the device files. This will update the LVM tables in memory.

# rmsf /dev/vg00/lvol9
# rmsf /dev/vg00/rlvol9

Now the VG displays normally (Cur LV is the same as Open LV).

# vgdisplay -v vg00
— Volume groups —
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 8
Open LV 8
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4384
VGDA 4
PE Size (Mbytes) 32
Total PE 8748
Alloc PE 1432
Free PE 7316
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

| Comments Off

Easy Xwindows for PCs

HP-UX

For the first couple of decades of Unix-style systems, Xwindows, also known as X or X11, was part of every Unix-based workstation. In HP-UX, Xwindows started life in the late 1980s with mwm (Motif Window Manager) and later in the early 1990s, porting VUE (Visual User Environment) from the recently acquired Apollo Domain/OS computer line. Later, HP adopted CDE (Common Desktop Environment) from the COSE (Common Open Software Environment) which is still part of HP-UX releases. Xwindows is a method to communicate graphical information to a display, called a display server.

When VUE or CDE is run on an HP-UX workstation , the client is running the Xwindow programs (VUE, CDE, xclock, etc) and the display server is the video card and display. But because Xwindows is a network protocol, the graphics can be sent to another computer, any computer that is running an Xwindows server program. Today, the demise of Unix-based workstations is almost complete with only Linux as the lone holdout. Unix-style servers are almost always managed with shell commands and more recently, web pages. Indeed, a web page is similar to Xwindows — the graphics and text is created by a web program, then presented to a browser which is really a display server for html code.

Starting with HP-UX 11.31, CDE is no longer started in a standard HP-UX installation. Most sysadmins manage their HP-UX systems using a terminal emulator and login using ssh or telnet. But CDE is a desktop manager and as such, takes over the entire screen. If you are using a PC running MS Windows, or even running Linux, you already have a local desktop with email and browsers already running. CDE overlays all of that so now you must switch desktops to return to your local programs. And very few improvements have been made to the toolset in CDE, so text editors, color and font managers and printing are primitive by today’s standards. Thus, CDE is now optional in the startup scripts.

But what if I need Xwindows for an Oracle installation or for Data Protector (xomni)?

No problem. These programs will run just fine without CDE. The CDE desktop is just that — a desktop with a few tools. You already have a desktop on a PC or Linux system. All that is needed is to run a Xwindow emulator on your PC (Linux has it running by default). You then connect to the HP-UX box and login normally, then assign and export the environment variable: DISPLAY with the IP address or hostname for your PC/Linux system, the display server in this case. Added to the hostname or IP address is the screen number starting from 0 (zero) and the optional workspace number as in:

export DISPLAY=1.2.3.4:0.0

for IP address 1.2.3.4 using display 0, workspace 0. Now you can run xclock and it will popup on your local desktop — no CDE required.

You can also run xterm, dtterm or hpterm. But that is not the best choice for logging into a Unix-based system. When you run xterm, etc, you are running a terminal emulator on the HP-UX system while your PC then interprets all the colors and graphics and fonts sent over the network. Typically, starting a single xterm requires hundreds of (very small) network packets just to display the first window. Additionally, fonts chosen by the Xclient may not match the fonts available in your local Xwindow emulator, with messages like: "Warning: Cannot convert…" or "Warning: Unable to load any usable…". The font size and/or family may not be available and the resultant display may not look as expected.

This can be mitigated by configuring your Xwindow emulator to use the fonts on your HP-UX system, thus guaranteeing a match. There are two steps for using a font server:

1. On your HP-UX system:
   1. edit /etc/rc.config.d/xfs and change RUN_X_FONT_SERVER=0 to =1
   2. Start the font server: /sbin/init.d/xfs start
2. On your local system, configure your Xwindow emulator to use a font server. This varies a lot between Xemulators but typically, you’ll specify the hostname or IP address and sometimes the port (7000) and protocol (TCP).

Now when you start an Xclient program such as xclock or xomni, you will see the correct fonts displayed and no error messages.

Xwindow clients for MS Windows can be expensive (Attatchmate/WRQ ‘s Reflection/X, OpenText (was Hummingbird) Exceed), or part of a bigger product (Cygwin/X, VMware), or freeware like Xming. By far, Xming is the smallest and simplest to use as long as you customize the startup. Download Xming from: http://sourceforge.net/projects/xming/ Once installed, you can start it using Xlaunch (no change to the first 2 windows, always check the box No Access Control on the last window), something like this:

I have tried specifying the font server in the last window but it doesn’t seem to work (hangs and can’t be stopped without using the Windows Task Manager). So I recommend using a command line approach:

1. Create a shortcut pointing to Xming (not Xlaunch).
2. Right-click the shortcut and select Properties
   Change the Target to:
   "C:\Program Files\Xming\Xming.exe" :0 -clipboard -multiwindow -ac -fp tcp/atl1:7000
   where atl1 is the hostname or IP address for the font server.
   Note: if the font server does not respond (wrong hostname, xfs not running, port 7000
   blocked, etc) then Xming will hang and must be killed with Task Manager.

Now Xming will start without any user interaction.

Refer to the SourceDirect blog:

http://blog.sourcedirect.com/?p=870

to see alternatives to the archaic Xwindow interface for terminal interaction.

| Comments Off

Network fails with /var full

HP-UX

One of the more bizarre error conditions for /var being full is that networking cannot be initialized at bootup. The basic networking functions (loopback interfaces) are the first step in getting network cards initialized, and if this step fails, all networking is disabled. All versions of HP-UX (10xx through 11.31) use net.init (found in /sbin/init.d) but if it fails, all subsequent network startup scripts will also fail. The after-boot symptoms will be:

• netstat -in shows no lan interfaces
• netstat -rn shows no routes
• all network access is disabled

You’ll only be able to access the system with the real console — and this is the way to differentiate between a ‘hang’ and failed networking. These error messages will be in /etc/rc.log:

Set privilege group …………………………………………. N/A
Display date ……………………………………………….. N/A
Copy processor logs to /var/tombstones …………………………
msgcnt 4 vxfs: mesg 001: vx_nospace – /dev/vg00/lvol8 file system full (1 block extent)
msgcnt 3 vxfs: mesg 001: vx_nospace – /dev/vg00/lvol8 file system full (1 block extent)

Configure Loopback interfaces (lo0) …………………………… FAIL *

VxVM volume recovery start …………………………………… OK

The code in net.init is trying to create a temporary file named: stcp.conf.$$ in /var/tmp or in /tmp.  Fix the problem in /var by searching for big directories (NOT big files). Here’s the commands:

# du -kx /var | sort -rn | head -20

Then look at the largest directories. /var/tmp is always a possible candidate for junk files that need to be removed. Similarly, /var/mail, /var/spool and /var/opt may need to be cleaned up. Once a few megabytes have been cleaned up, run this command:

# /sbin/init.d/net.init start

If you get nothing back, then networking will be started upon reboot.

| Comments Off

Here document not working

HP-UX

Here documents (inline text) are a great way to keep text within a script. But when the document doesn’t work as expected, troubleshooting can be a challenge. Here are some tips:

You can prevent shell expansion of variable names and other special characters by placing 2 "" (double quote) characters in front of the ending characters, like this:

cat << "" EOF
some text
some $VAR and special {}:"\|)(
EOF

Leading or trailing spaces (or tabs) will cause the shell to ignore the termination word (EOF in the above example). Make sure that the termination word (EOF in this case) starts in column1 and that there is nothing (including whitespace) following the word. When the here document is not terminated, the error messages will be very mysterious. Be sure to look at the first line number mentioned in the shell error message — it will be pointing to the start of the here document.

| Comments Off

PuTTY configs – setting things right

HP-UX

Many sysadmins that started on workstations or Linux will commonly use xterm, dtterm or even hpterm to connect to Unix-based systems. However, as modem users discovered many years ago, the massive amount of small packets needed to define the window and fonts was very painful. And even today, WAN connections where ping times are more than 75-100 ms are also very slow. The reason for Xwindow WAN delays (regardless of the wire speed) is the massive number of queries such as color, font and widget availability. Each query is a very short packet with a short response but the turnaround time (pure overhead) becomes very significant.

Rather than incur all that overhead in network (and trying to get Xwindows to work), you can use a terminal emulator that runs on your local PC. Now, the only network traffic will be the characters typed on the keyboard and those sent to the display. For PCs, the recommended program is PuTTY. It has all the functionality needed for managing Unix-style systems but with none of the overhead found in Xwindow applications such as xterm. And because it runs on the PC, you have all the tools for customization (colors, fonts, window resizing, etc) and presentation (printing, icons, multi-session collections). Best of all, PuTTY can be downloaded for free from:

    http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

The first thing to do before using PuTTY is to configure the options (and there are a lot of things that need fixing):

Start PuTTY and select: Session
Select Close window on exit: Always

Change to: Terminal -> Keyboard
Select Control-H so that the backspace key works.

Now select Window and change Columns = 132, Rows = 48.

It is a personal preference for changing rows/columns or changing the font size when resizing the window. Changing rows/columns is common, but changing font makes the text shrink/grow with the rows/columns staying the same size.

Change Lines of scrollback to 40000 (40k) so
you can see a lot of history.

Check:
Display
scrollbar,
Display in full screen mode
Reset scrollback on activity.

Select Window -> Appearance and check Block and Cursor blinks, Clear Type,
then.

Select Clear Type in Font quality.
NOTE: Old versions of PuTTY do not have Font quality.

Open the Change button.
Select Lucida
Console, Regular, 12 size.
If the font is too large, change to 9, 10 or 11

Windows -> Behavior and check just Window closes on ALT-F4

Select Windows -> Colours -> Default Foreground and change 187 to 255 (all 3).

Then select Default Background. Now it becomes user choice.
Dark colors are the best with white letters (255,255,255).
Dark brown, dark purple, dark green, dark orange are all good choices.

Now for creating additional logins, bring up the configuration window,
change the Host Name or IP address, and optionally change the background color.
Then type in a new name in Saved Sessions and press Save.

To help manage lots of predefined sessions, there are two session managers that are quite useful:

PuTTY Connection Manager — http://puttycm.free.fr/cms/

PuTTY Sessions Manager — http://puttysm.sourceforge.net/

| Comments Off

Troubleshooting failed NFS

HP-UX

There are always 2 parts to an NFS share: the client and the server. For shares that have been working in the past, the remote server and the path between is usually the problem. To see that if rpcbind is running and NFS transactions are getting through the network, use:

rpcinfo -p

You should see something like:

# rpcinfo -p
program vers proto   port  service
100000    4   tcp    111  rpcbind
100000    3   tcp    111  rpcbind
100000    2   tcp    111  rpcbind
100000    4   udp    111  rpcbind
100000    3   udp    111  rpcbind
100000    2   udp    111  rpcbind
100024    1   udp  49153  status
100024    1   tcp  49152  status
100133    1   udp  49153
100133    1   tcp  49152
100021    1   udp   4045  nlockmgr
100021    2   udp   4045  nlockmgr
100021    3   udp   4045  nlockmgr
100021    4   udp   4045  nlockmgr
100021    1   tcp   4045  nlockmgr
100021    2   tcp   4045  nlockmgr
100021    3   tcp   4045  nlockmgr
100021    4   tcp   4045  nlockmgr
1073741824    1   tcp  49153
100005    1   udp  49194  mountd
100005    1   tcp  49342  mountd
100005    2   udp  49194  mountd
100005    2   tcp  49342  mountd
100005    3   udp  49194  mountd
100005    3   tcp  49342  mountd
100003    2   udp   2049  nfs
100003    3   udp   2049  nfs
100227    2   udp   2049
100227    3   udp   2049
100003    2   tcp   2049  nfs
100003    3   tcp   2049  nfs
100227    2   tcp   2049
100227    3   tcp   2049

If a timeout error occurs, rpcbind is not running on this client, or some network filter such as IPFilter is blocking port 111 traffic.

Then, to check a remote server:

rpcinfo -p yoda

where yoda is the name of the remote NFS server:

# rpcinfo -p yoda
program vers proto   port  service
100000    4   tcp    111  rpcbind
100000    3   tcp    111  rpcbind
100000    2   tcp    111  rpcbind
100000    4   udp    111  rpcbind
100000    3   udp    111  rpcbind
100000    2   udp    111  rpcbind
100005    1   udp  49162  mountd
100005    3   udp  49162  mountd
100005    1   tcp  49295  mountd
100005    3   tcp  49295  mountd
100003    2   tcp   2049  nfs
100003    3   tcp   2049  nfs
100003    2   udp   2049  nfs
100003    3   udp   2049  nfs
100024    1   tcp  49296  status
100024    1   udp  49163  status
100021    1   tcp  49300  nlockmgr
100021    1   udp  49164  nlockmgr
100021    3   tcp  49301  nlockmgr
100021    3   udp  49165  nlockmgr
100021    4   tcp  49302  nlockmgr
100021    4   udp  49166  nlockmgr
100020    1   udp   4045  llockmgr
100020    1   tcp   4045  llockmgr
100021    2   tcp  49303  nlockmgr

Without a similar response (ie, a timeout), the client cannot communicate with the server’s rpcinfo process. rpcinfo uses port 111 and knows about the other rpc services such as rpc.lockd, rpc.statd and rpc.mountd. These services are required on the server for NFS to function. A timeout usually means that the port is blocked, whereas rpcinfo: can’t contact portmapper usually means that the service(s) are not running on the server. The error: rpcinfo: atldhcp3: n2a: hostname not found means that the hostname (or IP-addr) is invalid.

| Comments Off

Using lsof

HP-UX

lsof (List Open Files) is one of the most indispensible tools in the sysadmins toolkit. With it, you can look for processes that have opened a specific file or directory, find processes that have unlinked (removed from directory) files still open, find all the files opened by a specific process, or match network connections to processes.

What’s wrong with fuser? Unfortunately, it has not been updated for years and fails to find open files and directories about 50% of the time. lsof has not been part of a standard HP-UX installation so you may need to get a copy. It is very specific to the OS version. You can download a copy for 10.20 through 11.31 from:

    ftp://ftp.sourcedirect.com/pub/tools/lsof/

These are copied from either the lsof website, the Porting and Archive Centre for HP-UX

    (http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.84/)

or from HP’s Internet Express bundle.
These are standard self contained depots (except the 10.20 PA1.1 version which is just the program with no man page) so install like this:

    swinstall -s /tmp/lsof_whatever.depot \*

(don’t forget the \* at the end). No reboot is needed and there will be a man page for lsof.

Now for the good stuff. Here are examples of common usage for lsof:

File open: lsof /path_to/file

# lsof /etc/rc.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
PowerMoni 1012 root 1u REG 64,0×3 43729 1533 /etc/rc.log
PowerMoni 1012 root 2u REG 64,0×3 43729 1533 /etc/rc.log
xfs 1175 root 1u REG 64,0×3 43729 1533 /etc/rc.log
xfs 1175 root 2u REG 64,0×3 43729 1533 /etc/rc.log
ttd 1315 root 1u REG 64,0×3 43729 1533 /etc/rc.log
perfalarm 1342 root 7u REG 64,0×3 43729 1533 /etc/rc.log
perfalarm 1342 root 8u REG 64,0×3 43729 1533 /etc/rc.log
vxsvc 1637 root 1u REG 64,0×3 43729 1533 /etc/rc.log
vxsvc 1637 root 2u REG 64,0×3 43729 1533 /etc/rc.log
dtrc 1894 root 1u REG 64,0×3 43729 1533 /etc/rc.log
dtrc 1894 root 2u REG 64,0×3 43729 1533 /etc/rc.log

Directory open: lsof /path_to/directory

# lsof /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
PowerMoni 1012 root 5u unix 64,0×4 0t0 155 /tmp/s.ls3 (0x509d0c80)
postmaste 1068 sfmdb 5u unix 64,0×4 0t0 143 /tmp/.s.PGSQL.10864 (0x50c3b900)

Find unlinked file owner:
lsof -a +L1 /directory

# cd /tmp
# touch TObeREMOVED

# sleep 999 < TObeREMOVED &


# lsof TObeREMOVED

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sleep 5451 root 0u REG 64,0×4 0 209 TObeREMOVED



# rm TObeREMOVED

TObeREMOVED: ? (y/n) y
# ll TObeREMOVED

TObeREMOVED not found


# lsof -a +L1 /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep 5451 root 0u REG 64,0×4 0 0 209 /tmp (/dev/vg00/lvol4)

In the above example, the file TObeREMOVED is created, then sleep 999 is run with TObeREMOVED attached to the sleep process. While sleep is running in the background, TObeREMOVED is removed. However, the removal is a removal only from the directory listing (ie, unlinked). But the file still exists because sleep has the file open. Until sleep terminates (or is killed), the unnamed file still occupies space. lsof -a +L1 applied to the directory will find the open file. By stopping the program, the space can be released.

Network connections

Find all processes associated with a port 23 (telnet) connection: lsof -i tcp:23

# lsof -i tcp:23
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
inetd 1005 root 6u IPv6 0x42f443c0 0t0 TCP *:telnet (LISTEN)

Since the telnet port is managed by inetd (which spawns telnetd to perform the protocol), inetd is shown as the process using port 23. Use pstree to see the parent/child list for inetd:

# pstree 1005
-+- 01005 root /usr/sbin/inetd

|— 02328 root bootpd
|— 02295 root /etc/opt/resmon/lbin/registrar
\-+- 06158 root telnetd -b /etc/issue
\— 06159 blh -sh

NOTE: pstree is also available from: ftp://ftp.sourcedirect.com/pub/tools/pstree

Find port 22 (default Secure Shell) processes: lsof -i tcp:22

# lsof -i tcp:22
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 937 root 3u IPv6 0x509d0740 0t0 TCP *:22 (LISTEN)
sshd 937 root 4u IPv4 0x42f44040 0t0 TCP *:22 (LISTEN)
sshd 5088 root 3u IPv4 0x52054ac0 0t904536 TCP atl1:22->atldhcp30:3233 (ESTABLISHED)
sshd 6114 root 3u IPv4 0x52191c80 0t27882 TCP atl1:22->atldhcp30:3844 (ESTABLISHED)

As before, you can use pstree to see the process tree for a given PID (process ID):

# pstree 6114
-+- 06114 root sshd: root@pts/0
\-+- 06117 root -sh
\-+- 06287 root pstree 6114
\-+- 06288 root sh -c ps -ef
\— 06289 root ps -ef

Monitor (repeat) port 21 (ftp): lsof -i tcp:21 -r

# lsof -i tcp:21 -r
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
inetd 1005 root 5u IPv6 0x42f44200 0t0 TCP *:ftp (LISTEN)
=======
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
inetd 1005 root 5u IPv6 0x42f44200 0t0 TCP *:ftp (LISTEN)
ftpd 6354 root 0u IPv6 0x5227c900 0t241 TCP atl1:ftp->atl6:52021 (ESTABLISHED)
ftpd 6354 root 1u IPv6 0x5227c900 0t241 TCP atl1:ftp->atl6:52021 (ESTABLISHED)
=======
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
inetd 1005 root 5u IPv6 0x42f44200 0t0 TCP *:ftp (LISTEN)

Miscellaneous file listings:

Show open files for a given process: lsof -p 1234

# sleep 999 < /etc/issue &
[1] 6376


# lsof -p 6376
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sleep 6376 root cwd DIR 64,0×3 8192 2401 /root
sleep 6376 root txt REG 64,0×7 24576 732 /usr/bin/sleep
sleep 6376 root mem REG 64,0×7 1966080 33586 /usr/lib/libc.2
sleep 6376 root mem REG 64,0×7 24576 33983 /usr/lib/libdld.2
sleep 6376 root mem REG 64,0×7 393216 33982 /usr/lib/dld.sl
sleep 6376 root 0u REG 64,0×3 551 1872 /etc/issue
sleep 6376 root 1u STR 157,0 0t8563 1758 /dev/pts/0->ldterm->ptem->pts
sleep 6376 root 2u STR 157,0 0t8563 1758 /dev/pts/0->ldterm->ptem->pts

Show open files for all processes with a given name: lsof -c lvmkd

# lsof -c lvmkd
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
lvmkd 19 root cwd DIR 64,0×3 8192 2 /
lvmkd 20 root cwd DIR 64,0×3 8192 2 /

Show open files for a specific user: lsof -u blh

$ lsof -u blh
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 25268 blh cwd DIR 64,0×5 8192 16 /home/blh
sh 25268 blh txt REG 64,0×3 1469936 2646 /sbin/sh
sh 25268 blh 0u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
sh 25268 blh 1u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
sh 25268 blh 2u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
sh 25268 blh 27u REG 64,0×8 0 228 /var (/dev/vg00/lvol8)
sh 25268 blh 29uw REG 64,0×5 1240 24 /home/blh/.sh_history
lsof 25310 blh cwd DIR 64,0×5 8192 16 /home/blh
lsof 25310 blh txt REG 64,0×6 293112 43445 /opt/iexpress/lsof/bin/lsof
lsof 25310 blh mem REG 64,0×7 292876 15127 /usr/lib/hpux32/libxti.so.1
lsof 25310 blh mem REG 64,0×7 5021192 30401 /usr/lib/hpux32/libc.so.1
lsof 25310 blh mem REG 64,0×7 169484 8758 /usr/lib/hpux32/libnss_files.so.1
lsof 25310 blh mem REG 64,0×7 1407744 30829 /usr/lib/hpux32/libnsl.so.1
lsof 25310 blh mem REG 64,0×7 76240 8776 /usr/lib/hpux32/libdl.so.1
lsof 25310 blh mem REG 64,0×7 1148332 8775 /usr/lib/hpux32/dld.so
lsof 25310 blh mem REG 64,0×7 181240 8778 /usr/lib/hpux32/uld.so
lsof 25310 blh 0u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
lsof 25310 blh 1u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
lsof 25310 blh 2u STR 157,0 0t763245 983 /dev/pts/0->ldterm->ptem->pts
lsof 25310 blh 5w FIFO 0xe000000174053808 0t0 372575
lsof 25310 blh 6r FIFO 0xe000000174053088 0t0 372576

Show open files from an NFS import: lsof -N

# lsof -N
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 25150 root cwd DIR 4,0×3 1024 2 /mnt3
lsof 25231 root cwd DIR 4,0×3 1024 2 /mnt3

lsof -N is run from the client side to report on files that have been opened on this system from a remote NFS server.

| Comments Off