Using parallel processing for submits, syncs, and shelves

Submits and syncs

You can configure the server to allow parallel file transfer for submit and sync processing. Parallel processing is most effective with long-haul, high-latency networks, or with other network configurations that prevent the use of available bandwidth with a single TCP flow. In addition, parallel processing between a client running on a multi-core machine and the server allows the uncompression of large compressed binary files to happen in parallel on the client.

  • Use the net.parallel.max configurable to:

    • Transfer files in parallel during the submit process.
    • Speed up sync processing by having the p4 sync command transfer files using multiple threads. You do this by setting the net.parallel.max configuration variable to a value greater than one and by using the --parallel option to the p4 sync command.
  • Use the net.parallel.submit.threads configurable to specify the number of threads to be used for sending files in parallel for each submit (P4V 2017.3 and later).
  • Use the net.parallel.threads configurable to turn on parallel sync in a server. This parameter specifies the number of independent network connections that can be used for syncing files concurrently for each sync. When this parameter is set, parallel sync is automatically enabled in P4V as well (P4V 2017.3 and later).
  • To reduce lock contention during parallel syncs, set the client.sendq.dir configurable.

For more information, see the p4 submit and p4 sync commands in Helix Core P4 Command Reference.

Shelves

A p4 shelve command might execute more rapidly by transferring multiple files in parallel.

To enable automatic parallel shelving, set the net.parallel.shelve.threads configurable to a value that is less than or equal to the value of the net.parallel.max configurable.

Optionally, set the net.parallel.shelve.min configurable to change the minimum number of files in a parallel shelve, which, by default, is 9.

Optionally, set the net.parallel.shelve.batch configurable to change the number of files in a parallel shelve, which, by default, is 8.

To disable automatic parallel shelving, unset the net.parallel.shelve.threads configurable.

Note

Promoted shelves require an additional file transfer from the Edge to Commit Server. Parallel pull threads for this transfer are only used if the ExternalAddress field is set in its Edge Server spec and pull threads can be used on the Commit Server. This transfer using pull threads is currently not supported on Windows platforms.

A user can override the shelve configurables on the command line, or disable parallel shelving with the p4 shelve --parallel=0 command.