Users Guide to perfmerge

Contents

  1. Version Information
  2. What is perfmerge for?
  3. Terms and Conditions
  4. Preconditions for use
  5. Planning
  6. Limitations
  7. Pre-Merge Tasks
  8. The types of merge available
  9. Command Line Syntax
  10. Merging databases containing jobs
  11. Post-Merge tasks
  12. Examples

Version Information

This documentation covers perfmerge version 2023.2

What is perfmerge for?

Perfmerge is a tool for merging two Perforce repositories to produce a third repository. Unlike its predecessor, perfmerge does not work with checkpoints, rather it accesses the database files directly and creates new database files in the target directory. These new database files contain the merged content.

Terms and Conditions of Use

Perfmerge is provided in the hope that it will be useful, and all use is subject to the following Terms and Conditions:

  THIS SOFTWARE IS PROVIDED BY PERFORCE SOFTWARE, INC. 'AS IS' AND ANY
  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
  DISCLAIMED. IN NO EVENT SHALL PERFORCE SOFTWARE, INC. BE LIABLE FOR ANY
  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
  (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
  LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
  ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  

Perfmerge is a support utility, not a Perforce product. If you encounter any problems using this software, please contact support-helix-core@perforce.com.

Preconditions for use

Merging Perforce databases is a messy business: there are any number of incompatibilities that may exist between the databases being merged; perfmerge tests for the existence of some of these issues, but it is unlikely that perfmerge can cover all the issues. The most important pre-condition is the version of perfmerge:

You must ensure that the platform and release of perfmerge are identical on both databases to be merged. Using a newer perfmerge on your database may irreversibly upgrade your database tables necessitating a recovery from checkpoint.

Perfmerge will currently refuse to merge your databases if:

Planning

Perfmerge reads the complete contents of both databases (sets of db.* files). It is thus similar to performing a checkpoint. The time taken to perform a perfmerge is thus likely to be more or less the sum of the time taken to checkpoint each of the two source databases (in practice it may be a little less - but try and find out on your hardware!) It is therefore recommended to perform this on the fastest hardware you have available (both CPU and disks - so SSD drives will help!)

If you haven't restored your db.* files from a checkpoint recently, then performance may be improved by doing so before the merge. It is also recommended to do some house cleaning and remove old or unused client workspaces to reduce the size of db.have in the source databases.

Although perfmerge doesn't usually need archive files to do its job, (You will copy them in a post-merge step ) There is an exception if you have a //.swarm depot and are merging servers containing Swarm data. Before you merge you should verify that the swarm attachment archive files are accessable on each server by running "p4 verify -q //.swarm/attachments/..." .

Be careful when you copy the archive files to the merged server. Do not overwrite the .swarm depot on the merged server with the .swarm depot from one of the source servers. Doing so will cause the merged server to lose all of the swarm attachment archive file renames which were made during the merge.

Limitations

Merging Job Information

The facts that (a) jobs can be arbitrarily named, and (b) jobs can be based on a customized jobspec make the task of merging job information problematic. Perfmerge is able to merge job data from databases that use exactly the same jobspec, and that follow the default Perforce naming convention for their jobs (for example: job000001).

Job data from databases with differing jobspecs cannot be merged; however, you may choose to include only job information from one of the source databases in your merge result by using either of the '-jl' or '-jr' flags.

Jobs that are named using a local naming convention can be merged (assuming the jobspecs are identical), but they will not be renumbered: thus you should ensure that there are no job naming conflicts before merging.

Merging Server Configuration Data

With 2010.2 and later Perforce Servers, server configuration data set with the p4 configure command is stored within the Perforce database files in the db.config table. Note, server configuration data does not include counter data set using the p4 counter command. When merging databases that contain server configuration data, you must choose which database to take the server configuration data and history from by supplying the -cl or -cr option on the perfmerge command line. For example, to take the left database configuration data:

	$ perfmerge -t merge -cl /perforce/serverA /perforce/serverB /perforce/serverC

Server configuration data and history from the database not chosen will be written to the output database with its server name (P4NAME) value prepended with "left-" or "right-" accordingly. When merging databases that contain no server configuration data, the -cl or -cr option is not required. For example, if the configurations for the serverA and serverB databases in the example above were as follows:

    Left database (/perforce/serverA):

    $ p4 configure show allservers
    any: monitor = 2 
    any: P4LOG = master.log
    any: security = 1 
    any: server = 3
    replica1: db.replication = readonly
    replica1: lbr.replication = readonly
    replica1: P4LOG = replica.log
    replica1: P4TARGET = perforce:1666
    replica1: serviceUser = service
    replica1: startup.1 = pull -i 4
    replica1: startup.2 = pull -u -i 4
    replica1: startup.3 = pull -u -i 4

    Right database (/perforce/serverB):

    $ p4 configure show allservers
    any: monitor = 1
    any: security = 3 
    p4_1: net.mimcheck = 1 
    p4_1: P4LOG = log
    p4_1: server = 1  

The merged server configuration would look as follows if the -cl option was provided on the perfmerge command line to take the left database server configuration data:

    Merged database (/perforce/serverC):

    $ p4 configure show allservers
    any: monitor = 2
    any: P4LOG = master.log
    any: security = 1
    any: server = 3
    replica1: db.replication = readonly
    replica1: lbr.replication = readonly
    replica1: P4LOG = replica.log
    replica1: P4TARGET = perforce:1666
    replica1: serviceUser = service
    replica1: startup.1 = pull -i 4
    replica1: startup.2 = pull -u -i 4
    replica1: startup.3 = pull -u -i 4
    right-any: monitor = 1
    right-any: security = 3
    right-p4_1: net.mimcheck = 1
    right-p4_1: P4LOG = log
    right-p4_1: server = 1

See 'p4 help configure' for further details on managing server configuration data.

Merging User Data

If the servers being merged contain conflicting user data, perfmerge will produce the following error:

	perfmerge error:
		The servers being merged contain conflicting user names! Either resolve
		the conflicts or specify which database to take conflicting user data 
		from by supplying the -ul or -ur option on the perfmerge command line.

To correct the problem, either resolve the user conflicts manually by removing users from one of the input databases, or add the -ul or -ur option to the perfmerge command line to specify which input database to use when user conflicts are encountered. For example, if you have conflicting user data and want to take user data from the left input database when user conflicts exist, add the -ul option to the perfmerge command line:
	$ perfmerge -t merge -ul /perforce/serverA /perforce/serverB /perforce/serverC 

This instructs perfmerge to take user data from the left database (serverA in this case) when user conflicts exist. All non-conflicting user data will be merged into the output database. Any public keys stored by the server (p4 pubkeys) for a particular user are also affected by the -ul and -ur options; if you specify -ul and a user conflict exists, only public keys for that user in the left input database will be copied to the output database.

Merging Streams Data

With 2011.1 and later Perforce Servers, streams data may be present. When streams data is present, only offset merge types are supported. For example, attempting to run an append type merge against databases that contain streams data:

	$ perfmerge -t append /perforce/serverA /perforce/serverB /perforce/serverC

produces the following error:
	perfmerge error:
		The servers being merged contain streams data. Only offset type merges
		are supported when streams data is present.

Merging Unload Depots

With 2012.2 and later Perforce Servers, unload depots may be present. When unload depots are present, the naming of the unload depots and the presence of unloaded client or task stream data within unload depots may affect the merge process. The following rules apply:

Assuming an unload depot name of 'unload', the following commands can be used to produce a list of unloaded clients, labels, or streams:
	p4 files -U //unload/client/...

	p4 files -U //unload/label/...

	p4 files -U //unload/stream/...

For details on deleting or reloading clients, labels, or task streams, see the command reference for the p4 reload, p4 client, p4 label, and p4 stream commands.

Merging Servers with Swarm Data

perfmerge supports the merge of Swarm data using these options:

Merging right hand Swarm data requires an "offset" type merge

Merging swarm attachments requires access to the archive files, if the server has a //.swarm depot, please verify that "p4 verify //.swarm/attachments/..." runs cleanly on it before starting perfmerge

Merging Servers with Commons or Git Fusion Data

The presence of Commons, or Git Fusion data may affect the merge process. When Commons or Git Fusion data exists, merging is only allowed in the following cases:

Remote server specifications can be deleted from the input servers to remove the restrictions above as they pertain to remote specifications, but doing so will cause changelist renumbering to occur and prevent DVCS clients that had used 'p4 clone' against the input server(s) from using 'p4 push' and 'p4 fetch' to move data between their personal repos and merged server.

Merging Servers in a Distributed Perforce Environment

Merging servers that are part of a Commit/Edge Server architecture introduces restrictions. The following rules should be followed:

When only one of the input servers is a Commit Server, you can perform the merge without decommissioning Edge Servers following the steps below. An offset merge type is required and the Commit Server can't have its changelists renumbered so it must be specified as the left input database to the Perfmerge process. Note, before following the steps below take full checkpoints of the Edge and Commit Servers prior to carrying out the Perfmerge process. In the event the merge doesn't go as planned, this leaves you with a recovery point to fall back to.

1. Prevent new server activity on the Commit Server and wait for the Edge Server to catch up to the Commit Server in terms of metadata and archive replication
2. Shut down the Commit and Edge Server
3. Checkpoint the Edge Server
	$ cd $P4ROOT
	$ p4d -r . -k "db.have,db.working,db.resolve,db.locks,db.revsh,db.workingx,db.resolvex,db.view,db.label,db.revsx,db.revux" -jd edge.ckp

4. Use Perfmerge to merge the Commit Server using an offset merge type and the Commit Server as the left input database so no change renumbering occurs in that server
	$ perfmerge -t offset -o 10000 /commit/server/p4root /other/server/p4root /merged/server/p4root

5. Against the post-merged server, create a checkpoint to use for reseeding the Edge Server
	$ cd /merged/server/p4root
	$ p4d -r . -K "db.have,db.working,db.resolve,db.locks,db.revsh,db.workingx,db.resolvex" -z -jd commit-reseed-edge.ckp.gz  

6. On the Edge Server, replay the newly created post-merged Commit Server checkpoint and the Edge Server checkpoint from step 3 above into the edge P4ROOT
	$ cd $P4ROOT
	$ rm -rf db.*
	$ p4d -r . -z -jr commit-reseed-edge.ckp.gz
	$ p4d -r . -jr edge.ckp

7. Restart the Commit and Edge Server

Pre-Merge Tasks

It's a good idea to perform the following checks before performing the merge; in the end, it will save you time.

You should always, ALWAYS take a checkpoint and backup of both source databases before attempting a merge!

The Types of Merge Available

Perfmerge supports three different types of merge; they are all broadly similar and only differ in the way that changelists and jobs are renumbered as part of the merge.

Simple Merge

The simple merge is time-based: changelists from both databases are sorted by submit time, and then renumbered, starting from 1.

Jobs are sorted by jobname and by the order of the databases on the command line (i.e. if a job exists in both databases, the job from the first (left) database comes before the job from the second (right) database).

Append Merge

In an append merge, the changelists and jobs from the second database on the command line are 'appended' to those in the first database. This has the advantage of preserving the changelist numbers and job numbers from the first database, and the disadvantage that changelists no longer have their temporal nature. i.e. 'p4 files //...@<change>' can no longer be assumed to represent the state of your repository at a point in time.

Offset Merge

An offset merge is much like an append merge except that the changes and jobs from the second database are 'offset' from those in the first database. So, for example, if the first database had as its highest changelist change 14520, an offset merge might be used to append the second databases changes and jobs starting from 20000.

Summary Record options

Perfmerge, at the end of the process, writes a "summary record" change to the file //depot/perfmerge/map.txt. This file is always written to the last used change ID in the merged server. The change counter will always point to the next free change ID after the summary record (if any).

The "-S" option provides three options for writing this record. These options are explained in the following table:

Merge Type
"-t"
Merge Type Description -S "offset" or
no -S option specified
-S "next" -S "none"
simple Time Based merge.
Change IDs are "shuffled" into a
time based sequence.
A summary record is placed
at the next change id which is a
multiple of 1000
A summary record is placed
at the next change id.
No summary record is written.
append Change IDs from the second server
are appended to the first.
A summary record is placed
at the next change id which is a
multiple of 1000
A summary record is placed
at the next available change id.


No summary record is written.
offset Change IDs from second server
are started at an offset after the
end of the first. The offset is
specified using the "-o" option.
A summary record is placed
at 2 * offset beyond
the final change ID needed by
the merge.
A summary record is placed
at the next available change id.
No summary record is written.

Command Line Syntax

Synopsis

  perfmerge -t type [-o offset] [-C n] [-cl | -cr] [-jl | -jr]
            [-ul | -ur] [-v level ] [-V -h] [-S option]
            left-P4ROOT right-P4ROOT output-P4ROOT

      -t type     - Type of merge to perform. Supported merge types are:

                   	merge (simple, time-based merge)
                   	append
                   	offset

      -o offset   - For merge type offset, an additional offset value may
                    be specified.

      -C n        - Force case-sensitive (n=0), case-insensitive (n=1)
                    or 'hybrid' (n=2) ordering. Changing modes will
                    corrupt anything other than an empty database.
                    Note that all databases must use the same ordering.

      -cl         - Take server configuration data from left-P4ROOT.

      -cr         - Take server configuration data from right-P4ROOT.

      -h          - Help (this message)

      -jl         - Take job data only from left-P4ROOT. By default
                    job data is taken from both sources if they are
                    compatible.

      -jr         - Take job data only from right-P4ROOT. By default
                    job data is taken from both sources if they are
                    compatible.

      -S option  - Write Summary Change Options
                      'none' to disable writing summary change.
                      'next' to write the summary change at the
                           next available change ID.
                      'offset' to write the summary change at an offset
                           from the last change ID.
                    'offset' is the default behavior if -S is not used.

      -sl        - Take swarm data from the left.
      -sr        - Take swarm data from the right.
      -sb        - Merge swarm data from both sides.
      -sz        - Remove any swarm data.

      -ul         - Take conflicting user data from left-P4ROOT.
      -ur         - Take conflicting user data from right-P4ROOT.

      -v level    - Set verbosity level

      -V          - Print version and exit

Post-Merge Tasks

After merging there is always some tidying up to do. The minimum you should do is:

Note: Perfmerge 'submits' a change to the target database as part of the merge process, the change contains a single file (//depot/perfmerge/map.txt), the contents of which describe the way that the changelists and jobs were renumbered. You can use the information in this file to test the results of the merge to your satisfaction. Perfmerge prints the changelist number for this change in its output at the end of the merge process.

Examples

Simple merge

	$ mkdir /perforce/serverC
	$ perfmerge -t merge /perforce/serverA /perforce/serverB /perforce/serverC
  

This will merge the Perforce databases in /perforce/serverA and /perforce/serverB into the new Perforce database in /perforce/serverC. The merge will include jobs from both source databases providing the jobspecs are identical (otherwise it will fail).

Simple Merge with a single source for jobs

When merging databases with incompatible jobspecs, one option is to only take job information from one of the databases, and completely ignore any job information in the othe. Like this:

	$ mkdir /perforce/serverC
	$ perfmerge -t merge -jl /perforce/serverA /perforce/serverB /perforce/serverC
  

So in this case, the jobs will be taken from the left-hand source database (/perforce/serverA), and any jobs in /perforce/serverB will be ignored.

Append Merge

Say we want the jobs and changelists in serverA to be unchanged by the merge, but to merge in the contents of serverB. An append merge is just the ticket:

	$ mkdir /perforce/serverC
	$ perfmerge -t append /perforce/serverA /perforce/serverB /perforce/serverC
  

Offset Merge

Now let's suppose we want to be able to immediately see where merged changes came from when we look back at the history in the merged repository. In this case, an append merge doesn't quite fit the bill, and an offset merge is probably better:

	$ mkdir /perforce/serverC
	$ perfmerge -t offset -o 20000 /perforce/serverA /perforce/serverB /perforce/serverC
  

This will merge serverB with serverA, leaving serverA's change and job numbers unchanged, and appending serverB's changes and jobs starting with change/job 20000 (which must be greater than any existing change/job in serverA).

Document Info: $Id: //downloads/public/perforce/tools/perfmerge/perfmerge.html#20 $