User's Guide to perfsplit

Contents

  1. Version Information
  2. What is perfsplit for?
  3. Terms and Conditions
  4. Usage Cases
  5. Preconditions for use
  6. Changelog
  7. Pre-Split Tasks
  8. Command Line Syntax
  9. Splitmap Syntax
  10. Additional Flags
  11. Post-Split tasks
  12. Examples

Version Information

This documentation covers perfsplit versions up to and including 2023.1. This release binary can only be used to extract data from a 2023.1 Perforce server.

What is perfsplit for?

Perfsplit is a tool to extract a section of a depot from an existing Perforce database. It will access a Perforce Server's database directly, and is platform and release specific. Perfsplit will not remove data from the source Perforce Server.

Terms and Conditions of Use

Perfsplit is provided in the hope that it will be useful, and all use is subject to the following Terms and Conditions:

 Perfsplit is supplied by Perforce Software in the hope that it will be
useful. It is a support utility, not a Perforce product. All use of this
software is at the user's own risk and subject to the following terms and
conditions.

Copyright (c) 2020, Perforce Software, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL PERFORCE SOFTWARE, INC. BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Perfsplit is a support utility, not a Perforce product. If you encounter any problems using this software, please contact support-helix-core@perforce.com.

Usage Cases

Perfsplit has two distinct usage cases:

Preconditions for use

A splitmap is passed to 'perfsplit' that defines which paths should be included in the extraction. 'perfsplit' will perform some pre-split checks before extracting the appropriate information into a new P4ROOT directory, leaving the original database intact. Rather than define two broadly similar functions ('extract' and 'split'), this utility has but one function: extracting a sub-section of data from an existing Perforce database. Perfsplit will always extract mapped revision history but extraction of extra information; such as, working, label and fix information; can be specified by including one or more flags.

In addition to the database records, 'perfsplit' will copy the appropriate revisions from the depot files into a new P4ROOT. Please note, these files will be in depots under the new server's P4ROOT directory; therefore, local disk space is very important - there must be enough disk space for the database and all the file revisions that are being extracted.

Additional caveats:

Changelog

Major new functionality in 2023.1:

Bugs fixed in 2023.1:

None

Major new functionality in 2022.2:

None

Bugs fixed in 2022.2:

#114824
Core dump when map path includes escape sequence

Major new functionality in 2021.2:

None

Bugs fixed in 2021.2:

None

Major new functionality in 2022.1:

None

Bugs fixed in 2022.1:

None

Major new functionality in 2021.2:

None

Bugs fixed in 2021.2:

None

Major new functionality in 2021.1:

None

Bugs fixed in 2021.1:

None

Major new functionality in 2020.1:

Major new functionality in 2020.1:

None

Bugs fixed in 2020.1:

#103708
Bad type attr BSwchange! encountered using perfsplit with -A or -B option

Major new functionality in 2019.2:

None

Bugs fixed in 2019.2:

#98564
Missing db.desc entries are repaired

#98008
Checks for server.depot.root setting in the source server

Major new functionality in 2019.1:

#98914
Improved performance while extracting streams data

Major new functionality in 2018.2:

None

Major new functionality in 2018.1:

None

Major new functionality in 2017.2:

None

Major new functionality in 2017.1:

None

Major new functionality in 2016.2:

None

Major new functionality in 2016.1:

None

Major new functionality in 2015.2:

None

Major new functionality in 2015.1:

None

Major new functionality in 2014.2:

#945096
Add an option to extract temporary data from task streams.

#945073
Add an option to check database consistency at the end of the run.
This is no longer implied by the -A and -B options.

Bugs fixed in 2014.2:

#930449, #944186 (Bug #75471)
The -A and -B options no longer create database inconsistencies.

#906537 (Bug #74251)
Fix for db.workingx records not being split correctly.

#881880 (Bug #73428)
Do not require write access to the source database for archive files extraction.

#871421 (Bug #69710)
Allow spaces at the end of splitmap lines.

Major new functionality in 2014.1:

#839634 (Bug #73203)
Add long options support to Perfsplit.
Perfsplit now supports long options such as '--help' in addition to '-h'.
The usage info was updated with long options.

Bugs fixed in 2014.1:

#871479 (Bug #73206)
Skip properties which are specific to a particular user or group unless -Su is
specified.

#859328 (Bug #73038)
Avoid locking non-journaled tables, otherwise Perfsplit fails on Solaris.

#831720 (Bug #73205)
When using the -A and -B options, Perfsplit used to generate a jnl.fix file.
It is now saved as perfsplit.fix in the target directory, in case the current
directory already contains jnl.fix.

Major new functionality in 2013.2:

#735232 (Bug #49899)
New options to specify a start and an end change for the split. See documentation
about additional flags below.

Bugs fixed in 2013.2:

#661395, 677435 (Bug #66945)
Pre-split splitmap check can be limited by the PERFSPLIT_MAPLIMIT
environment variable. Splitmap entries beyond the limit are not tested for contents
and may not map any files.

The check could take an excessive amount of time when there are many lines in
the splitmap. By default, the check now stops after 1000 lines.

Major new functionality in 2013.1:

#636704, 61703 (Bug #61703, 66039)
For speed Persplit now extracts whole RCS files instead of rebuilding RCS
based on each revision that matches splitmap. This can lead to more revisions
in RCS file than can be accessed via Perforce. Using the '-l' flag will select the
old slower behavior.

#636704 (Bug #66039)
Copy centralsettings from the protects table if -Sd (server specs) option is selected.

629789 (Bug #66410)
Copy new db.nameval and db.property tables if -Sd (server specs) option is selected.

#611045 (Bug #59940)
Add Unicode support for Unicode enabled servers.

Bugs fixed in 2013.1:

#526751 (Bug #65681)
Admin protection table lines should be copied together with super lines. If the
path falls outside the split map, does not mean the user should lose privileges.

#629804 (Bug #56578)
 Shelve triggers should be filtered by path like change triggers.


Major new functionality in 2012.2:

#547232,545763 (Bug #61196)
 Skip any unloaded specs.  If a user wants to extract unloaded specs they
will  need to reload it first, then run perfsplit.

#541432 (Bug #59698, #60778)
Perfsplit will create the output directory if it does not exist. If the directory
does exist and it contains files Persplit will error before  the
Terms and
Conditions are displayed.

Bugs fixed in 2012.2:

#526751 (Bug #58143)
The split metadata no longer includes archmap records which are not in the split
map.


Major new functionality in 2012.1:

#491147
If the result contains '+X' files, tell the user that they need to copy the archive
script and trigger entries to the target server.

#491136
Changes to spec depot extraction:
    1. The spec depot is only extracted if the -Sp option is used or the split map
         includes a spec depot path (-SA implies -Sp).
    2. The specmap (new in 2012.1) is extracted with the spec depot.

#488842
Add a force option to skip librarian errors.

#488063
Additional tables copied:
   1. db.messages is now copied .
   2. db.nameval is now copied with db.counters and db.server is copied if
       server specs are requested.

Bugs fixed in 2012.1:

#529750 (Bug #51194, #59457)
'move/delete' revisions could cause perfsplit to skip extraction of librarian
revision. This has been fixed.

#522598 (Bug #59457)
'move/delete' revisions could cause archive extraction to fail with:
     "open for read: &ltlbrPath>: No such file or directory"
This has been fixed.


Major new functionality in 2011.1:

#386528, #386304, #384424 (Bug 48044)
Support stream extraction
2011.1 supports Streams, so perfsplit 2011.1+ should support extraction
from a streams depot.

#382951 (Bug 48207)
Log paramters used in output. This is useful for support if output has been
redirected to a file.

#382470 (Bug 48045)
Extracting only working information (-Sw) could result in shelved changelists
in the target server without any files attached. This has been resolved.

Pre-Split Tasks

It's a good idea to perform the following checks before performing the split; in the end, it will save you time.

Perfsplit can be run against a live server, but Perforce support recommend that a copy of the live system is restored to non production hardware.

IMPORTANT NOTE: Do not use perfsplit on a lower major version (for example 2009.2) of the server.

Command Line Syntax

Synopsis


        Usage: perfsplit [ options ] [ -Sx ] -r source -t target -m smap

           --root
           -r      path to source $P4ROOT

           --target
           -t      path to output $P4ROOT

           --mapfile
           -m      path to split map

        The -S/--split-flags option extends the core extraction to include:

           -Sb     branch specifications.
           -Sj     all jobs and relevant fixes.
           -Sl     mapped labels.
           -Sd     server specifications (servers, triggers, typemap, keys, properties).
           -Su     users, groups and protections.
           -Sc     client workspace specifications.
           -Sh     have list (implies -Su and -Sc).
           -Sp     spec depot (implied if split map includes spec depot paths)
           -Sw     opened files (implies -Su, -Sc, -Sj and -Sh).
           -Ss     shelved files (implies -Su, -Sc and -Sj ).
           -Sx     temporary data from task streams.

           --split-flags=A     implies all of the above.


        Where [options] are:

           --after=<n>
           -A <n>	restrict history to changes strictly after number n

           --upto=<n>
           -B <n>	restrict history to changes up to number n inclusive

           --force
           -f      skip archive files if there are errors

           --exact-archive
           -l      use old (low speed, by revision) archive file extraction

           --dbcheck
           -xx     Run a consistency check after creating the db files
                   The journal fixes are saved in perfsplit.fix

           --case-insensitive=<n>
           -C <n>  Force case-sensitive (n=0), case insensitive (n=1).
                   No other values are supported.

           --verbose=<n>
           -v <n>  set debug level to <n>

           --help
           -h      help (this message)

           --version
           -V      print version and exit

Splitmap Syntax

The splitmap defines the records that will be extracted from the source Perforce server. It is a list of paths that will be extracted using Perforce depot syntax. For example:

  //depot/folder1/...
//depot/folder2/...
//depot/folder3/...

Exclusionary mappings are also allowed. For example to exclude subfolders called 'test' from the mapping:

  //depot/folder1/...
//depot/folder2/...
//depot/folder3/...
-//depot/.../test/...

The 'spec' depot is treated as any other depot and can therefore be added to the splitmap. For example:

  //depot/folder1/...
//depot/folder2/...
//depot/folder3/...
//spec/...

Streams can be added to the splitmap but you may not specify paths under the stream. If one stream from a stream depot is found in the splitmap, all stream specs (definitions) for the streams depot will also be extracted. In fact, if not including all the paths in the stream hierachy the stream relationships may break. Streams paths and non streams paths can be used in the same splitmap file. For example:

  //depot/folder1/...
 //streams/mainline/...
//streams/rel1.0/...
//streams/rel2.0/...

NOTE:If there are spaces in the path, the line must be quoted. Comments must be on their own line and prefixed with '#'.

Additional flags

Perfsplit will always extract the following information based on the splitmap provided:

Extraction of additional information is controlled by the following flags:

Branch Specifications (--split-flags=b, -Sb)

Only branch specifications with a 'View' that contains paths listed in the splitmap are extracted. The 'View' is not filtered so may contain file paths that are not in the splitmap.

Jobs (--split-flags=j, -Sj)

All jobs will be extracted. Only fix records for extracted changelists will be extracted.

If the target server should only contain jobs that have associated fix records you can run a command to delete all jobs. Any job that has an associated fix record will fail to delete. For example from a *nix client:

  p4 jobs | cut -f1 -d" " | xargs -n1 p4 job -d 
and from a Windows client:
  for /F %i in ('p4 jobs') do p4 job -d %i

Labels (--split-flags=l, -Sl)

Any label that has a view that overlaps the splitmap will be extracted. However only files that are within the scope of the splitmap will be tagged. You may therefore extract over labels that had tagged files in the source but are empty in the target.

NOTE: For labels to be considered in the split, they must be reloaded before you begin the split. Labels that have 'autoreload' set must be converted to 'noautoreload' and reloaded before the split.

Other server specifications (--split-flags=d, -Sd)

This option copies the server specs, triggers and typemap tables, keys, properties and Swarm data. All 'change-*' triggers are interrogated and if the path is mapped in the splitmap, the trigger is extracted. For all other trigger types the trigger entry is extracted. Only typemap entries that are mapped in the splitmap are extracted.

NOTE: Trigger scripts are not extracted. These must be moved manually after the extraction.

Users, Groups and Protections (--split-flags=u, -Su)

All users and groups are extracted. Review entries attached to user records are filtered based on path. Only protection entries that contain paths that overlap the splitmap will be extracted. For safety, all 'super' entries are also extracted even if the path against the entry does not overlap the splitmap.

Client workspace Specifications (--split-flags=c, -Sc)

Only clients that have a 'View' that overlaps the splitmap will be extracted. Only 'View' entries that overlap the splitmap will be extracted.

NOTE: For client workspaces to be considered in the split, they must be reloaded before you begin the split.

Files stored in workspaces - have lists (--split-flags=h, -Sh)

The record of every synced revision that matches the splitmap will be extracted.

Spec depot definition (--split-flags=p, -Sp)

Copy the spec depot definition. To copy over archives from the spec depot, they still need to be included in the split map. Note that if the split map contains a spec depot path then this flag is implied and the spec depot is created.

Files opened (--split-flags=w, -Sw)

Any file that is open that matches the splitmap will be extracted. Only pending changelists that contain extracted pending revisions will be extracted.

If the source for a copied pending integration was outside the scope of the splitmap, you may see an error when using the extracted data. For example:

  Operation 'user-resolve' failed.
//depot/Jamgraph/MAIN/QAResults/jam-ndd.1.out is missing from the rev table!

To solve this, revert the pending integration

Shelved files (--split-flags=s, -Ss)

Any shelved file that matches the splitmap will be extracted. Only shelved changelists that contain extracted shelved revisions will be extracted.

Temporary data from task streams (--split-flags=x, -Sx)

Non-promoted revisions and integration records in task streams will be included in the extraction. This option is not required for extracting promoted data.

All data types (--split-flags=A, -SA)

Using the '-SA' flag is the same as specifying '--split-flags=bjlduchpwsx' or '-Sb -Sj -Sl -Sd -Su -Sc -Sh -Sp -Sw -Ss -Sx' in the perfsplit command line.

Debug levels (--verbose=<n>, -v <n>)

Persplit allows 3 debug levels:

Start and End change (--after=<n> and --upto=<n>, -A <n> and -B <n>)

These two flags limit the range of changelists to be extracted.
They are usually combined with the '-l' option to limit the size of RCS files at the expense of speed (the impact depends on the range).
The '-A' flag excludes all changes up to the specified change from the split. Note the specified change is also excluded.
The '-B' flag only includes changes up to the specified change in the split. Changes after it are not included.

Force (--force, -f)

Usually Persplit will abort if there are errors during archive extraction. Using this option will force Persplit++ to continue in these cases. To see filenames that had problems either use log level 1 or run 'p4 verify' against the extracted server.

Extract archive files by revision (--exact-archive, -l)

Use the old method (pre 2013.1) of extracting RCS revs one by one. This option is provided as a fall back if the new behavior of copying the whole RCS file is not desirable. For example if you have obliterated individual revisions from a file path for security reasons before passing to a 3rd party.
It can also be useful when using the -A and -B flags to limit the size of the archives in the split server.

Server Case Sensitivity (--case-insensitive=<n>, -C <n>)

If you are forcing case sensitivity using the '-C <n>' flag on source server startup, you will need to use the same flag for perfsplit. Using a different case sensitivity to the source server when running perfsplit is not supported and will raise an error.

Check database consistency (--dbcheck, -xx)

Run a consistency check at the end of metadata extraction.

If issues are found in the result database, it will output messages such as:

        Check db.have/db.rev...
        perfsplit error:
         	db.have/db.rev inconsistencies found.

In that case call Perforce Support to report the issue.
A file 'perfsplit.fix' gets created to correct the inconsistencies. Use the following command to apply the fix:
        p4d -r target -jr perfsplit.fix

Where 'target' is the path of the resulting server.

Post-Split Tasks

After extracting there is always some tidying up to do. The minimum you should do is:

After the extracted target server has been verified and is in use you can remove unwanted data from the source. For example:

NOTE: If you do not want users submitting to the path that was extracted, access can be excluded using the protections table.

To simplify the process of obliterating files in the source Perforce server perfsplit creates 'oblitlist.txt' in the same directory as the target 'db.*' files. This file contains a list of every file revision that was extracted. When you have confirmed that the correct files were extracted, 'oblitlist.txt' can be used as the input for a script to obliterate the extracted revisions from the source server.

NOTE: Always ensure you have a valid backup of database and depot files before obliterating data.

Examples

Simple split

  $ mkdir /perforce/serverB
$ perfsplit --root /perforce/serverA --target /perforce/serverB --mapfile splitmap.txt

This will extract from the Perforce database in /perforce/serverA and create a new Perforce database in /perforce/serverB. The target will include changelist, revision records, integration records, attributes attached to revisions and counters from the source database.

Simple split with tracing

  $ mkdir /perforce/serverB
$ perfsplit -v 3 -r /perforce/serverA -t /perforce/serverB -m splitmap.txt

This will extract from the Perforce database in /perforce/serverA and create a new Perforce database in /perforce/serverB, listing all files that were extracted and other debugging information.

Simple split with jobs

  $ mkdir /perforce/serverB
$ perfsplit -Sj -r /perforce/serverA -t /perforce/serverB -m splitmap.txt

In this case, all jobs will also be taken from the source database (/perforce/serverA).

Case insensitive split

For performance reasons, many Windows only client environments use Linux for the server operating system. To preserve the case insensitivity, the server flag '-C1' is used. If the source server is started with the '-C1' flag the split must also be case sensitive:

  $ mkdir /perforce/serverB
$ perfsplit -C1 -r /perforce/serverA -t /perforce/serverB -m splitmap.txt