Monitoring the Server

This chapter describes how you use p4d commands to monitor the server and its resources:

  • You use the p4 diskspace command to monitor diskspace usage.
  • You use the p4 monitor command to monitor processes.
  • You set server trace flags with the p4d startup command to diagnose problems.
  • You examine logs to track commands that exceed predetermined thresholds of resource usage.
  • You use the p4 monitor command to display information about locked files.
  • You use the P4AUDIT environment variable to enable the auditing of file access.
  • You use a variety of logging commands to log information and manage log files.

Monitoring disk space usage

Use the p4 diskspace command to monitor diskspace usage. By default, p4 diskspace displays the amount of free space, diskspace used, and total capacity of any filesystem used by Perforce.

By default, the Perforce Server rejects commands when free space on the filesystems housing the P4ROOT, P4JOURNAL, P4LOG, or TEMP fall below 10 megabytes. To change this behavior, set the filesys.P4ROOT.min (and corresponding) configurables to your desired limits:

Configurable Default Value Meaning

filesys.P4ROOT.min

10M

Minimum diskspace required on server root filesystem before server rejects commands.

filesys.P4JOURNAL.min

10M

Minimum diskspace required on server journal filesystem before server rejects commands.

filesys.P4LOG.min

10M

Minimum diskspace required on server log filesystem before server rejects commands.

filesys.TEMP.min

10M

Minimum diskspace required for temporary operations before server rejects commands.

filesys.depot.min

10M

Minimum diskspace required for any depot before server rejects commands. (If there is less than filesys.depot.min diskspace available for any one depot, commands are rejected for transactions involving all depots.)

If the user account that runs the Perforce Server process is subject to disk quotas, the Server observes these quotas with respect to the filesys.*.min configurables, regardless of how much physical free space remains on the filesystem(s) in question. The next section explains the options you have in reconfiguring default values.

Specifying values for filesys configurables

In specifying filesys.*.min values, you have the option of specifying an absolute number or a percentage indicating a portion of the current space. So, there are six possible numeric formats you can use, as shown in the following table:

Format Meaning

nnn

A plain number, used as is.

nnnK

A number in kilobytes

For example, the following command sets filesys.P4TEMP.min to 100 kilobytes. 

$ p4 configure set filesys.P4TEMP.min=100K

nnnM

A number in megabytes

For example, the following command sets filesys.P4ROOT.min to 10 megabytes. 

$ p4 configure set filesys.P4ROOT.min=10M

nnnG

A number in gigabytes.

For example, the following command sets filesys.P4JOURNAL.min to 1 gigabytes. 

$ p4 configure set filesys.P4JOURNAL.min=1G

nnnT

A number in terabytes.

nnn%

A number as a percentage of the current space.

For example, the following command means that at least ten percent of the total disk space must be free and available for P4ROOT. 

$ p4 configure set filesys.P4ROOT.min=10%

Determining available disk space

To estimate how much disk space is currently occupied by specific files in a depot, use the p4 sizes command with a block size corresponding to that used by your storage solution. For example, the command: 

$ p4 sizes -a -s -b 512 //depot/...

shows the sum (-s) of all revisions (-a) in //depot/..., as calculated with a block size of 512 bytes. 

//depot/... 34161 files 277439099 bytes 5429111 blocks

The data reported by p4 sizes actually reflects the diskspace required when files are synced to a client workspace, but can provide a useful estimate of server-side diskspace consumption.

Monitoring processes

Use the p4 monitor command to observe and control Perforce-related processes running on your Perforce server machine.

The following sections explain how you enable process monitoring and list running processes,.

Enabling process monitoring

Server process monitoring requires minimal system resources, but you must enable process monitoring for p4 monitor to work. To monitor all active commands, set the monitor configurable as follows: 

$ p4 configure set monitor=1

Additional settings offer more options:

  • 0: Server process monitoring off. (Default)
  • 2: monitor both active commands and idle connections.
  • 5: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second.
  • 10: monitor both active commands and idle connections, including a list of the files locked by the command for more than one second, with lock wait times included in the lock information.
  • 25: monitor both active commands and idle connections, including a list of the files locked by the command for any duration, with lock wait times included in the lock information.

How you set up monitoring levels 5, 10, and 25, depends on the platform where the server is running. See the description of the p4 monitor command in P4 Command Reference for more information.

Enabling idle processes monitoring

By default, IDLE processes (often associated with custom applications based on the Perforce API) are not included in the output of p4 monitor. To include idle processes in the default output of p4 monitor, use monitoring level 2. 

$ p4 configure set monitor=2

To display idle processes, use the command:

$ p4 monitor show -s I

Listing running processes

To list the processes monitored by the Perforce server, use the command:

$ p4 monitor show

To restrict the display to processes currently in the running state, use the command: 

$ p4 monitor show -s R

By default, each line of p4 monitor output looks like this: 

pid status owner hh:mm:ss command [args]

where pid is the UNIX process ID (or Windows thread ID), status is R or T depending on whether the process is running or marked for termination, owner is the Perforce user name of the user who invoked the command, hh:mm:ss is the time elapsed since the command was called, and command and args are the command and arguments as received by the Perforce server. For example: 

$ p4 monitor show
74612 R qatool     00:00:47 job
78143 R edk        00:00:01 filelog
78207 R p4admin    00:00:00 monitor

To show the arguments with which the command was called, use the -a (arguments) flag: 

$ p4 monitor show -a
74612 R qatool     00:00:48 job job004836
78143 R edk        00:00:02 filelog //depot/main/src/proj/file1.c //dep
78208 R p4admin    00:00:00 monitor show -a

To obtain more information about user environment, use the -e flag. The -e flag produces output of the form: 

pid client IP-address status owner workspace hh:mm:ss command [args]

where client is the Perforce application (and version string or API protocol level), IP-address is the IP address of the user’s Perforce application, and workspace is the name of the calling user’s current client workspace setting. For example: 

$ p4 monitor show -e
74612 p4/2011.1 192.168.10.2   R qatool    buildenvir 00:00:47 job
78143           192.168.10.4   R edk       eds_elm    00:00:01 filelog
78207 p4/2011.1 192.168.10.10  R p4admin   p4server   00:00:00 monitor

By default, all user names and (if applicable) client workspace names are truncated at 10 characters, and lines are truncated at 80 characters. To disable truncation, use the -l (long-form) option: 

$ p4 monitor show -a -l
74612 R qatool     00:00:50 job job004836
78143 R edk        00:00:04 filelog //depot/main/src/proj/file1.c //dep
ot/main/src/proj/file1.mpg
78209 R p4admin    00:00:00 monitor show -a -l

Only Perforce administrators and superusers can use the -a, -l, and -e options.

Setting server trace and tracking flags

To modify the behavior of command tracing or performance tracking, specify the appropriate -v subsystem=value flag to the p4d startup command. Use P4LOG or the -L `logfile` flag to specify the log file. For example: 

$ p4d -r /usr/perforce -v server=2 -p 1666 -L /usr/perforce/logfile

Before you activate logging, make sure that you have adequate disk space.

Note

When running Perforce as a Windows service, use the p4 set command to set P4DEBUG as a registry variable. You can also set these trace flags when running p4d.exe as a server process from the command line.

Setting server debug levels on a Perforce server (p4d) has no effect on the debug level of a Perforce Proxy (p4p) process, and vice versa.

Higher levels of the Perforce server command tracing and tracking flags are typically recommended only for system administrators working with Perforce Technical Support to diagnose or investigate problems.

Command tracing

The server command trace flags and their meanings are as follows.

Trace flag Meaning

server=0

Disable server command logging.

server=1

Logs server commands to the server log file.

(As of release 2011.1, this is the default setting.)

server=2

In addition to data logged at level 1, logs server command completion and basic information on CPU time used. Time elapsed is reported in seconds. On UNIX, CPU usage (system and user time) is reported in milliseconds, as per getrusage().

server=3

In addition to data logged at level 2, adds usage information for compute phases of p4 sync and p4 flush (p4 sync -k) commands.

For command tracing, output appears in the specified log file, showing the date, time, username, IP address, and command for each request processed by the server.

Performance tracking

The Perforce Server produces diagnostic output in the server log whenever user commands exceed certain predetermined thresholds of resource usage. Performance tracking is enabled by default, and if P4DEBUG is unset (or the tracking flag is not specified on the command line), the tracking level is computed based on the number of users in the license file.

Tracking flag Meaning

track=0

Turn off tracking.

track=1

Track all commands.

track=2

Track excess usage for a server with less than 10 users.

track=3

Track excess usage for a server with less than 100 users.

track=4

Track excess usage for a server with less than 1000 users.

track=5

Track excess usage for a server with more than 1000 users.

The precise format of the tracking output is undocumented and subject to change.

Showing information about locked files

You can use the -L option of the p4 monitor to show information about locked files. The information is collected only for the duration of the p4 monitor command and is not persisted. See the description of the p4 monitor command for more information about how to set up this kind of monitoring.

The following sample output to the p4 monitor show -L command, shows the information displayed about locked files: 

8764 R user 00:00:00 edit
      [server.locks/clients/88,d/ws4(W),db.locks(R),db.rev(R)]
8766 R user 00:00:00 edit
      [server.locks/clients/89,d/ws5(W),db.locks(R),db.rev(R)]
8768 R user 00:00:00 monitor

Following pid, status, owner, and time information, output shows two edit commands that have various files locked, including the client workspace lock in exclusive mode for the workspaces ws4 and ws5, and db.locks and db.rev tables in read-only mode.

Auditing user file access

The Perforce Server is capable of logging individual file accesses to an audit logfile. Auditing is disabled by default, and is only enabled if P4AUDIT is set to point to the location of the audit log file, or if the server is started with the -A auditlog flag.

When auditing is enabled, the server adds a line to the audit log file every time file content is transferred from the server to the client. On an active server, the audit log file will grow very quickly.

Lines in the audit log appear in the form:

date time user@client clientIP command file#rev

For example:

$ tail -2 auditlog
2011/05/09 09:52:45 karl@nail 192.168.0.12 diff //depot/src/x.c#1
2011/05/09 09:54:13 jim@stone 127.0.0.1 sync //depot/inc/file.h#1

If a command is run on the machine that runs the Perforce Server, the clientIP is shown as 127.0.0.1.

If you are auditing server activity in a replicated environment, each of your build farm or forwarding replica servers must have its own P4AUDIT log set.

Logging and structured log files

The Perforce Server can be configured to write log files in a structured (.csv) format. Structured log files contain more detail than conventional log files, are easier to parse, and the Perforce Server offers additional commands to help customize your site’s logging configuration. This section summarizes the commands you use to manage logging, describes the use of structured logs, and explains how log files are rotated.

Note

All p4d error and info logs are in UTF8 for a server in unicode mode. You need an UTF8 console or editor to properly render this log information.

Logging commands

You can use the following commands to work with logs.

Command Meaning

p4 logappend

If the user log is enabled, write an entry to user.csv

p4 logparse

Parse a structured log file and return the logged data in tagged format

p4 logrotate

Rotate a named logfile, or, if no name is specified, all server logs. This command applies only to structured logs; it does not rotate the unstructured P4LOG or P4AUDIT logs.

p4 logstat

Report the file size of the journal (P4JOURNAL, error log (P4LOG), audit log (P4AUDIT), or the named structured log file.

p4 logtail

Output the last block of the error log (P4LOG)

p4 logschema

Return a description of the specified log record type.

Enabling structured logging

To enable structured logging, set the serverlog.file.N configurable to the name of the file. Valid names for structured log files and the information logged are shown in the following table. You can use a file path in conjunction with the file name.

Warning

You must use one of the file names specified in the table. If you use an arbitrary name, no data will be logged to the file you specify.

Filename Description

all.csv

All loggable events (commands, errors, audit, etc…​)

audit.csv

Audit events (audit, purge)

auth.csv

The results of p4 login attempts. If the login failed, the reason for this is included in the log. Additional information provided by the authentication method is also included.

commands.csv

Command events (command start, compute, and end)

errors.csv

Error events (errors-failed, errors-fatal)

events.csv

Server events (startup, shutdown, checkpoint, journal rotation, etc.)

integrity.csv

Major events that occur during replica integrity checking.

route.csv

Log the full network route of authenticated client connections. Errors related to net.mimcheck are also logged against the related hop.

track.csv

Command tracking (track-usage, track-rpc, track-db)

user.csv

User events; one record every time a user runs p4 logappend.

Files do not have to be set in consecutive order; for example, this is fine: 

$ p4 configure set serverlog.file.1=audit.csv
$ p4 configure set serverlog.file.2=auth.csv
$ p4 configure set serverlog.file.4=track.csv

Note

Enabling all structured logging files can consume considerable diskspace. See Structured logfile rotation for information on how to manage the size of the log file and the number of log rotations.

The value you specify for N may not exceed 500.

Structured logfile rotation

Each of the configured serverlog.file.N files has its own corresponding serverlog.maxmb.N and serverlog.retain.N configurables. For each configured server log type, these configurables control the maximum size (in megabytes) of the logfile before rotation, and the number of rotated server logs retained by the server.

Structured log files are automatically rotated on checkpoint, journal creation, overflow of associated serverlog.maxmb.N limit (if configured), and the p4 logrotate command. You can disable log rotation after journal rotation by setting the configurable dm.rotatelogwithinjnl to 0. Disabling this behavior can help when you’re doing frequent journal rotations and you want the log rotated on a different schedule.

You can use the serverlog.counter.N configurable to create a counter that tracks the number of times a structured log file has been rotated. For example, the following command creates a rotation counter called myErrorsCount: 

$ p4 configure set serverlog.counter.3=myErrorsCount

Each time the errors.csv log file is rotated, the counter is increased by one. In addition, the name of the log file is changed to specify the pre-incremented counter value. That is, if the counter myErrors is 7, the errors.csv file is named errors-6.csv.

You can create a counter for each file described in the preceding table. Do not use system reserved counter names for your counter: change, maxCommitChange, job, journal, traits, upgrade.

The p4 logtail command returns the current value of the counter when you logtail that log. It also returns the current size of the log at the end of the output (along with the ending offset in the log). The size and offset are identical if p4 logtail reads to the end of the log. Security monitoring tools can use counters and the p4 logtail command in the process of scanning log files to monitor suspicious activity.