User's Guide to perfsplit

Contents

Version Information

This documentation covers perfsplit versions up to and including 2024.1. This release binary can only be used to extract data from a 2024.1 Perforce server.

What is perfsplit for?

Perfsplit is a tool to extract a section of a depot from an existing Perforce database. It will access a Perforce Server's database directly, and is platform and release specific. Perfsplit will not remove data from the source Perforce Server.

Terms and Conditions of Use

Perfsplit is provided in the hope that it will be useful, and all use is subject to the following Terms and Conditions:

 Perfsplit is supplied by Perforce Software in the hope that it will be
 useful. It is a support utility, not a Perforce product. All use of this
 software is at the user's own risk and subject to the following terms and
 conditions.

 Copyright (c) 2020, Perforce Software, Inc.  All rights reserved.

 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

 1.  Redistributions of source code must retain the above copyright
     notice, this list of conditions and the following disclaimer.
 2.  Redistributions in binary form must reproduce the above copyright

    notice, this list of conditions and the following disclaimer in the

    documentation and/or other materials provided with the distribution.


    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"

    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE

    ARE DISCLAIMED. IN NO EVENT SHALL PERFORCE SOFTWARE, INC. BE LIABLE FOR ANY

    DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

    (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

    LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND

    ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF

    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Perfsplit is a support utility, not a Perforce product. If you encounter any problems using this software, please contact support-helix-core@perforce.com.

Usage Cases

Perfsplit has two distinct usage cases:

• Extracting data from the main server. For example, to extract revision history for a project to pass to a third party.
• Splitting the Perforce server into two separate Perforce servers. In this case perfsplit can be run to extract data to the target server, then unwanted data can be obliterated/deleted from the source server. Alternatively a second split can be performed to extract the paths not already extracted.

Preconditions for use

A splitmap is passed to 'perfsplit' that defines which paths should be included in the extraction. 'perfsplit' will perform some pre-split checks before extracting the appropriate information into a new P4ROOT directory, leaving the original database intact. Rather than define two broadly similar functions ('extract' and 'split'), this utility has but one function: extracting a sub-section of data from an existing Perforce database. Perfsplit will always extract mapped revision history but extraction of extra information; such as, working, label and fix information; can be specified by including one or more flags.

In addition to the database records, 'perfsplit' will copy the appropriate revisions from the depot files into a new P4ROOT. Please note, these files will be in depots under the new server's P4ROOT directory; therefore, local disk space is very important - there must be enough disk space for the database and all the file revisions that are being extracted.

Additional caveats:

• Cannot be used to extract data from graph depots (depots of type 'graph').
• Will not undo lazy copies - this can be done before using 'p4 snap'.
• Will not extract revisions from a remote depot.
• Will not extract depot spec for remote depot.
• Will not extract to a remote machine. Target P4ROOT must be local to the source database.
• Will not create a checkpoint of the target database.
• Will not read the environment for P4ROOT.
• Depots will not be extracted to a path outside the target P4ROOT.
• Will not support overlay mappings in a split map.
• If a path in a streams depot is selected for extraction every stream specification for that stream depot will also be extracted.
• Will not copy SSL certificates from the source.
• Will not copy Server ID.
• Will not extract unloaded clients/labels or 'autoreload' labels.

Changelog

None

• Added support for db.revtr.

• Added support for db.revfs.

None

None

#114824
Core dump when map path includes escape sequence

None

None

None

None

None

None

None

None

• Will copy db.upgrades to target server.
• Will re-populate db.storage on target by running upgrade "BuildStorage191"
• Will re-populate db.storagesx and db.storagesh on target by running upgrade "BuildShelfStorageIndex"

None

#103708
Bad type attr BSwchange! encountered using perfsplit with -A or -B option

None

#98564
Missing db.desc entries are repaired

#98008
Checks for server.depot.root setting in the source server

#98914
Improved performance while extracting streams data

None

None

None

None

None

None

None

None

#945096
Add an option to extract temporary data from task streams.

#945073
Add an option to check database consistency at the end of the run.
This is no longer implied by the -A and -B options.

#930449, #944186 (Bug #75471)
The -A and -B options no longer create database inconsistencies.

#906537 (Bug #74251)
Fix for db.workingx records not being split correctly.

#881880 (Bug #73428)
Do not require write access to the source database for archive files extraction.

#871421 (Bug #69710)
Allow spaces at the end of splitmap lines.

#839634 (Bug #73203)
Add long options support to Perfsplit.
Perfsplit now supports long options such as '--help' in addition to '-h'.
The usage info was updated with long options.

#871479 (Bug #73206)
Skip properties which are specific to a particular user or group unless -Su is
specified.

#859328 (Bug #73038)
Avoid locking non-journaled tables, otherwise Perfsplit fails on Solaris.

#831720 (Bug #73205)
When using the -A and -B options, Perfsplit used to generate a jnl.fix file.
It is now saved as perfsplit.fix in the target directory, in case the current
directory already contains jnl.fix.

#735232 (Bug #49899)
New options to specify a start and an end change for the split. See documentation
about additional flags below.

#661395, 677435 (Bug #66945)
Pre-split splitmap check can be limited by the PERFSPLIT_MAPLIMIT
environment variable. Splitmap entries beyond the limit are not tested for contents
and may not map any files.

The check could take an excessive amount of time when there are many lines in
the splitmap. By default, the check now stops after 1000 lines.

#636704, 61703 (Bug #61703, 66039)
For speed Persplit now extracts whole RCS files instead of rebuilding RCS
based on each revision that matches splitmap. This can lead to more revisions
in RCS file than can be accessed via Perforce. Using the '-l' flag will select the
old slower behavior.

#636704 (Bug #66039)
Copy centralsettings from the protects table if -Sd (server specs) option is selected.

629789 (Bug #66410)
Copy new db.nameval and db.property tables if -Sd (server specs) option is selected.

#611045 (Bug #59940)
Add Unicode support for Unicode enabled servers.

#526751 (Bug #65681)
Admin protection table lines should be copied together with super lines. If the
path falls outside the split map, does not mean the user should lose privileges.

#629804 (Bug #56578)
 Shelve triggers should be filtered by path like change triggers.

#547232,545763 (Bug #61196)
 Skip any unloaded specs.  If a user wants to extract unloaded specs they
will  need to reload it first, then run perfsplit.

#541432 (Bug #59698, #60778)
Perfsplit will create the output directory if it does not exist. If the directory
does exist and it contains files Persplit will error before  the Terms and
Conditions are displayed.

#526751 (Bug #58143)
The split metadata no longer includes archmap records which are not in the split
map.

#491147
If the result contains '+X' files, tell the user that they need to copy the archive
script and trigger entries to the target server.

#491136
Changes to spec depot extraction:
    1. The spec depot is only extracted if the -Sp option is used or the split map
         includes a spec depot path (-SA implies -Sp).
    2. The specmap (new in 2012.1) is extracted with the spec depot.

#488842
Add a force option to skip librarian errors.

#488063
Additional tables copied:
   1. db.messages is now copied .
   2. db.nameval is now copied with db.counters and db.server is copied if
       server specs are requested.

#529750 (Bug #51194, #59457)
'move/delete' revisions could cause perfsplit to skip extraction of librarian
revision. This has been fixed.

#522598 (Bug #59457)
'move/delete' revisions could cause archive extraction to fail with:
     "open for read: &ltlbrPath>: No such file or directory"
This has been fixed.

#386528, #386304, #384424 (Bug 48044)
Support stream extraction
2011.1 supports Streams, so perfsplit 2011.1+ should support extraction
from a streams depot.

#382951 (Bug 48207)
Log paramters used in output. This is useful for support if output has been
redirected to a file.

#382470 (Bug 48045)
Extracting only working information (-Sw) could result in shelved changelists
in the target server without any files attached. This has been resolved.

Pre-Split Tasks

It's a good idea to perform the following checks before performing the split; in the end, it will save you time.

• Check for sufficient disk space available to save the extracted data. This will be no more than the disk space used for the source of the extraction.
• If performing a split, disallow write access for all users to the area to be split.
• Optionally use 'p4 snap' to undo lazy copies. The main use case is when splitting out a depot that has files integrated in from other depots.
  
• Download the build of perfsplit for that Perforce version from the Perforce Web/FTP Site
• Create the target directory. Note that perfsplit will not run if the target directory already contains 'db.*' files.

Perfsplit can be run against a live server, but Perforce support recommend that a copy of the live system is restored to non production hardware.

Command Line Syntax

Synopsis

        Usage: perfsplit [ options ] [ -Sx ] -r source -t target -m smap

           --root
           -r      path to source $P4ROOT

           --target
           -t      path to output $P4ROOT

           --mapfile
           -m      path to split map

        The -S/--split-flags option extends the core extraction to include:

           -Sb     branch specifications.
           -Sj     all jobs and relevant fixes.
           -Sl     mapped labels.
           -Sd     server specifications (servers, triggers, typemap, keys, properties).
           -Su     users, groups and protections.
           -Sc     client workspace specifications.
           -Sh     have list (implies -Su and -Sc).
           -Sp     spec depot (implied if split map includes spec depot paths)
           -Sw     opened files (implies -Su, -Sc, -Sj and -Sh).
           -Ss     shelved files (implies -Su, -Sc and -Sj ).
           -Sx     temporary data from task streams.

           --split-flags=A     implies all of the above.


        Where [options] are:

           --after=<n>
           -A <n>	restrict history to changes strictly after number n

           --upto=<n>
           -B <n>	restrict history to changes up to number n inclusive

           --force
           -f      skip archive files if there are errors

           --exact-archive
           -l      use old (low speed, by revision) archive file extraction

           --dbcheck
           -xx     Run a consistency check after creating the db files
                   The journal fixes are saved in perfsplit.fix

           --case-insensitive=<n>
           -C <n>  Force case-sensitive (n=0), case insensitive (n=1).
                   No other values are supported.

           --verbose=<n>
           -v <n>  set debug level to <n>

           --help
           -h      help (this message)

           --version
           -V      print version and exit

Splitmap Syntax

The splitmap defines the records that will be extracted from the source Perforce server. It is a list of paths that will be extracted using Perforce depot syntax. For example:

  //depot/folder1/...
  //depot/folder2/...
  //depot/folder3/...
 

Exclusionary mappings are also allowed. For example to exclude subfolders called 'test' from the mapping:

  //depot/folder1/...
  //depot/folder2/...
  //depot/folder3/...
  -//depot/.../test/...
 

The 'spec' depot is treated as any other depot and can therefore be added to the splitmap. For example:

  //depot/folder1/...
  //depot/folder2/...
  //depot/folder3/...
  //spec/...
 

Streams can be added to the splitmap but you may not specify paths under the stream. If one stream from a stream depot is found in the splitmap, all stream specs (definitions) for the streams depot will also be extracted. In fact, if not including all the paths in the stream hierachy the stream relationships may break. Streams paths and non streams paths can be used in the same splitmap file. For example:

  //depot/folder1/...
  //streams/mainline/...
  //streams/rel1.0/...
  //streams/rel2.0/...

NOTE:If there are spaces in the path, the line must be quoted. Comments must be on their own line and prefixed with '#'.

Additional flags

Perfsplit will always extract the following information based on the splitmap provided:

• counters and configurables
• revision records
• integration records
• archived revision records
• changelists
• attributes
  
• all stream specs for any stream depot included in the splitmap
  

Branch Specifications (--split-flags=b, -Sb)

Only branch specifications with a 'View' that contains paths listed in the splitmap are extracted. The 'View' is not filtered so may contain file paths that are not in the splitmap.

Jobs (--split-flags=j, -Sj)

All jobs will be extracted. Only fix records for extracted changelists will be extracted.

If the target server should only contain jobs that have associated fix records you can run a command to delete all jobs. Any job that has an associated fix record will fail to delete. For example from a *nix client:

  p4 jobs | cut -f1 -d" " | xargs -n1 p4 job -d 
      

  for /F %i in ('p4 jobs') do p4 job -d %i
      

Labels (--split-flags=l, -Sl)

Any label that has a view that overlaps the splitmap will be extracted. However only files that are within the scope of the splitmap will be tagged. You may therefore extract over labels that had tagged files in the source but are empty in the target.

NOTE: For labels to be considered in the split, they must be reloaded before you begin the split. Labels that have 'autoreload' set must be converted to 'noautoreload' and reloaded before the split.

Other server specifications (--split-flags=d, -Sd)

This option copies the server specs, triggers and typemap tables, keys, properties and Swarm data. All 'change-*' triggers are interrogated and if the path is mapped in the splitmap, the trigger is extracted. For all other trigger types the trigger entry is extracted. Only typemap entries that are mapped in the splitmap are extracted.

NOTE: Trigger scripts are not extracted. These must be moved manually after the extraction.

Users, Groups and Protections (--split-flags=u, -Su)

All users and groups are extracted. Review entries attached to user records are filtered based on path. Only protection entries that contain paths that overlap the splitmap will be extracted. For safety, all 'super' entries are also extracted even if the path against the entry does not overlap the splitmap.

Client workspace Specifications (--split-flags=c, -Sc)

Only clients that have a 'View' that overlaps the splitmap will be extracted. Only 'View' entries that overlap the splitmap will be extracted.

NOTE: For client workspaces to be considered in the split, they must be reloaded before you begin the split.

Files stored in workspaces - have lists (--split-flags=h, -Sh)

The record of every synced revision that matches the splitmap will be extracted.

Spec depot definition (--split-flags=p, -Sp)

Copy the spec depot definition. To copy over archives from the spec depot, they still need to be included in the split map. Note that if the split map contains a spec depot path then this flag is implied and the spec depot is created.

Files opened (--split-flags=w, -Sw)

Any file that is open that matches the splitmap will be extracted. Only pending changelists that contain extracted pending revisions will be extracted.

If the source for a copied pending integration was outside the scope of the splitmap, you may see an error when using the extracted data. For example:

  Operation 'user-resolve' failed.
  //depot/Jamgraph/MAIN/QAResults/jam-ndd.1.out is missing from the rev table!

Shelved files (--split-flags=s, -Ss)

Any shelved file that matches the splitmap will be extracted. Only shelved changelists that contain extracted shelved revisions will be extracted.

Temporary data from task streams (--split-flags=x, -Sx)

Non-promoted revisions and integration records in task streams will be included in the extraction. This option is not required for extracting promoted data.

All data types (--split-flags=A, -SA)

Using the '-SA' flag is the same as specifying '--split-flags=bjlduchpwsx' or '-Sb -Sj -Sl -Sd -Su -Sc -Sh -Sp -Sw -Ss -Sx' in the perfsplit command line.

Debug levels (--verbose=<n>, -v <n>)

Persplit allows 3 debug levels:

• 1 - (Default) Displays summary of each step being run.
• 3 - Displays summary of each step and each database tables as it is processed.
• 5 - Displays summary of each step, database tables as they are processed, number of records that are extracted from each table and list of file revisions that are extracted. All maps used within the extraction are displayed. This level is intended for debugging and support.

Start and End change (--after=<n> and --upto=<n>, -A <n> and -B <n>)

These two flags limit the range of changelists to be extracted.
They are usually combined with the '-l' option to limit the size of RCS files at the expense of speed (the impact depends on the range).
The '-A' flag excludes all changes up to the specified change from the split. Note the specified change is also excluded.
The '-B' flag only includes changes up to the specified change in the split. Changes after it are not included.

Force (--force, -f)

Usually Persplit will abort if there are errors during archive extraction. Using this option will force Persplit++ to continue in these cases. To see filenames that had problems either use log level 1 or run 'p4 verify' against the extracted server.

Extract archive files by revision (--exact-archive, -l)

Server Case Sensitivity (--case-insensitive=<n>, -C <n>)

If you are forcing case sensitivity using the '-C <n>' flag on source server startup, you will need to use the same flag for perfsplit. Using a different case sensitivity to the source server when running perfsplit is not supported and will raise an error.

Check database consistency (--dbcheck, -xx)

Run a consistency check at the end of metadata extraction.

If issues are found in the result database, it will output messages such as:

        Check db.have/db.rev...
        perfsplit error:
         	db.have/db.rev inconsistencies found.

        p4d -r target -jr perfsplit.fix

Post-Split Tasks

After extracting there is always some tidying up to do. The minimum you should do is:

• Run 'p4d -r . -xx' and 'p4d -r . -xv' in the target server P4ROOT.
• If you need a new server ID or need to recreate the existing ID use 'p4 serverid'.
  
• Create/Edit your protections table to provide the security needed for the new server.
• Run a 'p4 verify' in the new target to check that the depot files are in the correct location.
• Check your groups and their membership.
• Check your typemap.
• Remove unwanted stream specs from any streams depots extracted.
  
• Describe a sample of changelists in the target server and check the results against the original database.
• Take a checkpoint and a full backup before using the split database!
• Optionally restore the database files from checkpoint. Perfsplit inserts rows into some database files in a different order to their index. Restoring from checkpoint will correctly order these tables and may decrease the database file sizes.
• Move/copy the depot files and new database tables to the new physical server.
• If system is SSL enabled regenerate certificates. See the Administration Guide for additional details.
  
• Check that the paths in the depot specifications point to the correct depot file location on the server disk.
• Copy triggers scripts to new server and test.
• Recreate tickets for all batch processes - tickets will be invalid after the change in server information.

After the extracted target server has been verified and is in use you can remove unwanted data from the source. For example:

• Obliterate revision that were split into the new target.
• Remove changelists that are now empty.
• Remove stream specs for revisions that were split into the new target.
• Update protections table if necessary.
• Check your groups and their membership.

NOTE: If you do not want users submitting to the path that was extracted, access can be excluded using the protections table.

To simplify the process of obliterating files in the source Perforce server perfsplit creates 'oblitlist.txt' in the same directory as the target 'db.*' files. This file contains a list of every file revision that was extracted. When you have confirmed that the correct files were extracted, 'oblitlist.txt' can be used as the input for a script to obliterate the extracted revisions from the source server.

NOTE: Always ensure you have a valid backup of database and depot files before obliterating data.

Examples

Simple split

  $ mkdir /perforce/serverB
  $ perfsplit --root /perforce/serverA --target /perforce/serverB --mapfile splitmap.txt
  

This will extract from the Perforce database in /perforce/serverA and create a new Perforce database in /perforce/serverB. The target will include changelist, revision records, integration records, attributes attached to revisions and counters from the source database.

Simple split with tracing

  $ mkdir /perforce/serverB
  $ perfsplit -v 3 -r /perforce/serverA -t /perforce/serverB -m splitmap.txt
  

This will extract from the Perforce database in /perforce/serverA and create a new Perforce database in /perforce/serverB, listing all files that were extracted and other debugging information.

Simple split with jobs

  $ mkdir /perforce/serverB
  $ perfsplit -Sj -r /perforce/serverA -t /perforce/serverB -m splitmap.txt
  

In this case, all jobs will also be taken from the source database (/perforce/serverA).

Case insensitive split

For performance reasons, many Windows only client environments use Linux for the server operating system. To preserve the case insensitivity, the server flag '-C1' is used. If the source server is started with the '-C1' flag the split must also be case sensitive:

  $ mkdir /perforce/serverB
  $ perfsplit -C1 -r /perforce/serverA -t /perforce/serverB -m splitmap.txt