Users Guide to p4migrate

The current version of p4migrate is 2023.1. This supports all servers up to and including 2023.1.
Using a Perforce Server greater than 2023.1 with p4migrate may mean you see errors in the conversion and is not supported

Contents

  1. What is p4migrate for?
  2. The Problems
  3. The Solution
  4. Where to download?
  5. Notes and Limitations
  6. Migrating from Windows to Unix
  7. Migrating from Unix to Windows
  8. Archive File Conversion Issues
  9. Contacting Support

What is p4migrate for?

p4migrate is a tool that works with a checkpoint of your Perforce database in order to facilitate migrating your Perforce server between platforms.

p4migrate will help you generate a case-consistent checkpoint, by default this checkpoint will be marked as case sensitive. The p4migrate -C1 option allows you to generate a case insensitive checkpoint instead.

The Problems

Windows and OSX are case-insensitive, Unix is case-sensitive. Similarly, Perforce on Windows and OSX is case-insensitive and Perforce on Unix is case-sensitive. This is because Perforce relies heavily on the semantics of the underlying filesystem on which your data is stored.

Case-Consistent Meta-data

As Perforce on Windows is case-insensitive, the Perforce meta-data on a Windows server may contain references to files whose names differ only by case. On Windows these are references to the same file, however if you migrate meta-data in this state to Unix, they will be seen as references to different files.

This situation arises because Windows users are not always consistent in their use of case when referring to files and is called a case inconsistency.

As Unix is case sensitive, Unix users have little choice but to be consistent in their use of case if they want their commands to work so the same problem does not arise on the Unix platforms.

Archive Filename Case

On Windows servers more recent than 2000.2, the Perforce server consistently uses lowercase names for the files and directories that make up the revision archives. This is not the meta-data (which is in the db.* files), but the depot directories in which the physical copies of your file revisions reside.

When you migrate to Unix, these archive files/directories must be renamed to match the case in your meta-data or the Perforce server will be unable to locate any files or revisions.

The Solution

p4migrate solves both of the problems above. It first helps you to produce a case-consistent copy of your meta-data by editing a checkpoint according to rules you define; and then it can rename the archive files/directories to match the case in this edited checkpoint.

Note that it's not perfect. There are some types of case-inconsistency that are not possible to detect computationally as they may be intentional. It's unlikely, but they may. These p4migrate leaves for the user to resolve after the migration is complete. They all have very low impact and are easy to resolve.

So when you move a Perforce installation from Windows to Unix, there are essentially three steps to the process:

  1. Create a case-consistent checkpoint
  2. Rename the archive files/dirs to match the checkpoint
  3. Correct the line endings in the archive files

Using p4migrate

Introduction

Before getting started with p4migrate, some definitions are useful

Term Explanation
conflict map A mapping (in the Perforce sense) that describes the locations of case-inconsistency within your meta-data. The conflict map is produced by p4migrate from an analysis of your checkpoint. The conflict map contains one mapping for each inconsistency found.
translation map A mapping (in the Perforce sense) supplied as input to p4migrate. All paths in your checkpoint are translated through this mapping before undergoing conflict analysis or being written to the output checkpoint. Typically an edited conflict map is used as the translation map.

Example Conflict/Translation Maps

It's worth looking at a couple of example conflict/translation maps in more detail. Remember a conflict map is what p4migrate generates, a translation map is a conflict map after you've edited it to express your preferences.

So, if p4migrate detects that you have been inconsistent in your use of case when referring to files in the //depot/main/www/... directory, it will generate a conflict map entry to help you resolve this conflict.

The left hand side of the mapping is in mixed case and is the source pattern

The right hand side of the mapping is in mixed case and is the destination pattern.

The mappings are case-insensitive on all platforms.

Initially both sides are identical and are set to the most commonly used variant of the path with conflicts. You change the right hand side only.

So in the example above, the conflict mapping produced by p4migrate might look like this:


    //depot/main/WWW/... //depot/main/WWW/...

So in this case, p4migrate offers you the //depot/main/WWW/... variant as your default because that is the most frequently observed variant in the checkpoint file. If that's what you want, then you're ready to proceed, but if not then you should edit the file and change the case (and only the case) to suit your preferences. Then the file is ready to be used as a translation map.

What if you'd been inconsistent in your use of the "main" directory too? Well in that case the mappings might have looked like this:


    //depot/Main/...  //depot/Main/...
    //depot/Main/WWW/...  //depot/Main/WWW/...

So in this case, if you're unhappy with the case of the "Main" directory, you must change it in the right hand side of both mapping lines.

Conflict/Translation Map Rules

The mappings in p4migrate conflict/translation maps are just plain old Perforce mappings. This means that there are two golden rules:

And some p4migrate specific rules

In practical terms this means that you must

or p4migrate will not work as expected. The second rule is tough to understand so an example is in order. Say p4migrate generates a mapping like this:

  //depot/QA/Gastar/... //depot/QA/Gastar/...

and you edit it before using it as a translation map and you alter the case of the QA directory as well as changing the case of the leaf-directory (Gastar) where the conflict actually occurred.

  //depot/QA/Gastar/... //depot/qa/gastar/...

In this case, you must also add a new mapping for the QA directory to your translation map or you will have introduced a new conflict all by yourself for //depot/QA/... paths other than the Gastar subdirectory.

  //depot/QA/... //depot/qa/...
  //depot/QA/Gastar/... //depot/qa/gastar/...

Note that p4migrate just appends new conflict mappings to the existing translation map so in some circumstances p4migrate itself may generate mappings that are not in the correct order. Therefore you should always review your mappings carefully.

Right

  //depot/production/... //depot/production/...
  //depot/production/gastar/... //depot/production/gastar/...
  //depot/qa/... //depot/qa/...
  //depot/qa/gastar/... //depot/qa/gastar/...

Wrong

  //depot/production/gastar/... //depot/production/gastar/...
  //depot/qa/gastar/... //depot/qa/gastar/...
  //depot/production/... //depot/production/...
  //depot/qa/... //depot/qa/...

Using Locales

p4migrate has 3 locale based configurations, configured by the -L locale setting

  1. If the locale is set to either 'C' or 'POSIX', p4migrate will use high performance case folding intended for 7bit characters.
  2. If the locale is set to a value which contains the substring 'UTF-8', then a generic utf8 case folding algorithm will be used.
  3. Otherwise, assuming that the locale specified is installed on your machine, case folding will use the (slow) locale specific case folding code available in the libc libraries

If the checkpoint is from a unicode mode server then utf8 specific case folding will be used.

if no -L locale option is passed, the 'C' locale will be used by default

There is a chance, that for a given locale setting, p4migrate will miss some case folding conversions. If this happens, multiple permutations of a given case insensitive path may still exist in the resulting checkpoint. You may wish to try the migration again using a different locale setting to see if it works better.

Using Policies

Policies can be used to apply a global transformation to all paths in your repository. If it's your policy that all filenames should be lower- or upper-case, then p4migrate can correct all exceptions to this policy and bring all your pathnames into line.

  p4migrate -p upper -c conflicts -o checkpoint.new checkpoint.NNNN
  p4migrate -p lower -c conflicts -o checkpoint.new checkpoint.NNNN

There's also a policy called 'head' which will not transform the paths in any way, but rather affects the path that will be chosen for inclusion in the conflict map. Instead of choosing the most frequently-used case-variant, it will select the case used by the head revision (in the case of conflicts) at the filename level and, for conflicts at the folder level, it will choose the most recently encountered variant used by the head revisions of the files underneath that folder.

  p4migrate -p head -c conflicts -o checkpoint.new checkpoint.NNNN

There are some caveats with the use of policies:

  1. p4migrate will use the configured locale to change the case of paths
  2. Your checkpoint may contain paths in different character sets, so they may not all share the same correct 'locale' settings
  3. a checkpoint containing mixed file paths in both utf-8 and locale specific character sets may not convert correctly using any 'locale' setting

Advanced Policies Features

You can override your policy using the Translation Map (-t transmap) to specify the exceptions to the rule as the Translation Map is applied after the policy. When you do this, ensure that you provide a complete set of mappings and use the -c flag with p4migrate so that any conflicts (perhaps caused by missing mappings) are recorded in a conflicts map so that you can correct your translation map and try again. For example:

        p4migrate -p upper -c conflicts -t transmap -o checkpoint.new checkpoint.NNNN
 

Note, using this feature you will need to confirm that you have understood that this can casue unexpected results if applied incorrectly.

The "Translation Map" is in the same format of the conflict maps. Any path that is mentioned in the the Translation Map will be applied after the policy. In the above the policy is "upper". If the Translation Map contains:

         //depot/production/... //DEPOT/production/...
         //depot/production/gastar/... //DEPOT/production/gastar/...
 

The the paths starting with "//DEPOT/production/..." and "//DEPOT/production/gastar/..." will all be in lower case matching the translation map, and all other depot will be in upper case Matching the policy, upper. For example, your final depot layout would appear as:

     //DEPOT/FOO.TXT
     //DEPOT/DEV/FOO.TXT
     //DEPOT/production/LIVE/FOO.TXT
     //DEPOT/production/gastar/QA/FOO.TXT
 

Applying policies and Translation Maps at the same time can lead to unexpected results if applied incorectly. If in doubt, contact Perforce Technical Support for assistance.

Command Line Syntax

  Usage: p4migrate [options] <checkpoint file>

  where options are:

    -c <filename>      - Filename to use for the output translation map,
                         also known as the 'conflict map'.

                         Conflicts in your checkpoint are recorded in this
                         file by generated mappings from the lowercase
                         variant of the filename to the most frequently
                         observed variant in your checkpoint. You should
                         edit this file and change the mappings to meet
                         your requirements and then supply the edited file
                         as the input translation map using the -t flag.

    -C0                - Generate a case sensitive checkpoint (default)

    -C1                - Generate a case insensitive checkpoint

    -j                 - Produce the output as a journal patch rather
                         than a replacement checkpoint.

    -l <filename>      - Filename to use for logfile. Defaults to stderr.

    -L <locale>        - The locale to use during the migration.
                         Not used if the source checkpoint is from a 'unicode' server.
                         The 'C' locale is set by default, and is the fastest for migrating,
                         but will not support case folding of multi-byte characters.
                         Any other locale may be specified, but it may slow down processing,
                         and will require that the specified locale is installed on the machine.
                         If the locale specified contains the string 'UTF-8' then utf8
                         case folding will be enabled.

    -o <filename>      - Filename to use for the output checkpoint.
                         After mapping records through the input translation
                         map, the edited records are written out to this
                         file. If no further conflicts remain, then this is
                         a case-consistent checkpoint. If conflicts remain
                         unresolved then this checkpoint should not be used
                         rebuild your database. You may pass '-' to use
                         stdout.

    -p <policy>        - Name of a built-in policy to apply. Valid policies
                         are:

                            lower    - convert all paths to lowercase
                            upper    - convert all paths to uppercase
                            head     - convert all paths to the variant used
                                       by the head revision

                         Note: The values for upper and lowercase characters
                         are very dependent on your locale. If your
                         checkpoint contains characters in different
                         character sets you should exercise great care with
                         the use of this option.

                         In particular, you should run 'p4 verify -q //...' 
                         and check the output very carefully.

                         Note 2: Policies are applied before the translation
                         map, so you may override the policy for specific
                         paths using the translation map. You will likely
                         need several mappings to get this right. e.g.
                         with a policy of 'upper' you might use something
                         like this:

                             //depot/...          //Depot/...
                             //depot/main/...     //Depot/Main/...
                             //depot/main/www/... //Depot/Main/www/...

                         i.e. a mapping for every part of the tree that
                         you want to exempt from the policy


    -R                 - Attempt to rename the archive files to match the
                         case in the translated checkpoint. This option
                         is only available on case-sensitive platforms.

    -r <p4root>        - Your P4ROOT directory, this option may be needed if -R is used,
                         and the server.depot.root configurable is not found.

    -s                 - Will convert the input checkpoint to
                         'standard checkpoint' format.

    -S <serverId>      - The ServerId for this server, this may be required when
                          server.depot.root configurable is found.

    -t <filename>      - Filename for the input translation map.
                         The paths in your checkpoint are remapped through
                         the mappings in this file to resolve conflicts
                         due to inconsistent use of case. Default: none
                         See also -c.

    -u                 - Source checkpoint is from unicode mode server.
                         This option is only needed if the checkpoint is missing a
                         Checkpoint header journal note.

    -v <level>         - Output verbosity, (0-6) higher levels provide more detail

    -V                 - Print version details and exit.

Where to download p4migrate and documentation?

  1. Download p4migrate documentation from here: http://ftp.perforce.com/perforce/tools/p4-migrate/p4migrate.html
  2. Download p4migrate for your target machine from the same place you'd look for 'p4d', in: "http://ftp.perforce.com/perforce/RELEASE/ARCHITECTURE/p4migrate/".

    For instance the 64 bit linux release of r20.2 can be found here: http://ftp.perforce.com/perforce/r20.2/bin.linux26x86_64/p4migrate

    Unlike other support tools, p4migrate works with any previous release of perforce, you don't need to exactly match your p4d version

Additional Notes and Limitations

Standard checkpoint format

Often when diffing checkpoints there are a many differences because of optional quoting around integer values. The -s option will convert the input checkpoint into a 'standard checkpoint' format to allow textual diffs to show content changes cleanly.

 p4migrate -s -o checkpointOut checkpointIn

will read checkpointIn and output it to checkpointOut in standard checkpoint format

Renaming

If the -R flag is used, p4migrate will attempt to rename the archive files to match the given checkpoint. It uses the following algorihm to find the archive files base directory:

  1. If the -r <p4root> option is used, then this value is used as the base directory for the archive files
  2. If the -r <p4root> option is not used, p4migrate will look in db.config for a configurationVersion which provides a default serverId. The default serverId can be overridden by using the "-S <serverId>" command option.
  3. If a server.depot.root entry in db.config is found which matches our serverId then its value will be used as the base directory for the archive files.

Groups

Group names used in properties and permissions are renamed to match the case of existing db.group and db.groupx table entries.

Archive Depots

Archive depots names will automatically be converted to lower case. This is to maintain consistency with the rev records. If you have files that have been archived using the "p4 archive" command, the depot path for the archived files will be consistent with the translation map. For example, if your translation map is as follows:

  //depot/production/spares/... //depot/Production/Spares/...
And your archive Depot is called BUCKET, the migrated paths will become:
 //bucket/depot/Production/Spares/...

When renaming the archive files as described below, it is advised that you have your archive depot attached to the P4ROOT. Failing to do so will mean that P4Migrate will not be able to rename any archived files and you will need to do so manually before restoring the archive files.

Stream Depots and Streams

Stream Depots and Streams are treated in the same way as other file names within P4Migrate. The output within the cmap file may appear similar to:

  //STRDepot/Prod_650P_DEV //STRDepot/Prod_650P_DEV
//STRDepot/Prod_650P_DEV/... //STRDepot/Prod_650P_DEV/...
With //STRDepot/Prod_650P_DEV as the name of the Stream and //STRDepot/Prod_650P_DEV/... the files underneath the Stream. The same rules that apply to non-stream file paths apply to both the Stream name and path.

Unloaded Data

Currently p4migrate does not support unloaded data as this contains meta data which is written to disk in the archive location which may not be available to p4migrate. This metadata may contain case inconsistent data and as such should be reloaded to make sure that all any case issues in this data are resolved. Before you create your checkpoint to test, check the output of:

  p4 labels -U
  p4 clients -U
  p4 streams -U

Migrating from Windows to Unix

This is the most common migration and the one which p4migrate was primarily designed to support; to move from a case-insensitive platform and run a case-sensitive Perforce Server on a Unix based platform. If you are wanting to run a case-insensitive Perforce Server on a Unix based platform, then you should use the p4d "-C1" flag as p4migrate is not required for this purpose. Please contact Perforce Support for further information on this flag.

Pre-Migration tasks

  1. Make sure that the depot Map: paths are either relative to P4ROOT or server.depot.root (if configured) or changed to the new absolute location for the target platform. If the Map: needs to be changed, you should do so after starting the Perforce Server on localhost. This can be done as follows:

    1. Shutdown the Perforce Server by running:
          p4 admin stop
          
    2. Restart the Perforce Server listening only on localhost:1666
          p4d -r P4ROOT -p localhost:1666
          
    3. Connect to the Perforce Server specifying the localhost:1666 port and change the depot Map:
          p4 -p localhost:1666 login
      p4 -p localhost:1666 depot depot
    4. Change depot Map: be relative or be the full path for the location of the archive files on your Perforce Server. For example, a relative depot Map: path:
          Map:	depot/...
          
      and an absolute path:
          Map:	/perforce/archives/depot/...
          
    5. Repeat this for all depots on your Perforce Server
    6. Shutdown the Perforce Server
          p4 -p localhost:1666 admin stop
          

Migration tasks

  1. Take a fresh checkpoint and a full backup. Do not be tempted to skip this step!
  2. Run a "p4 verify -q //..." to ensure the integrity of your archive files.
  3. Copy the checkpoint created, the archive files and license file to your target machine.
    1. Download p4migrate for your target machine from the same place you'd look for 'p4d', in: "ftp://ftp.perforce.com/perforce/RELEASE/ARCHITECTURE/p4migrate/".
    2. For instance the 64 bit linux release of r22.1 is here: ftp://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4migrate.
    3. unlike other support tools, p4migrate works with any previous release of perforce, you don't need to exactly match your p4d version
  4. Run p4migrate to identify the case-conflicts within your checkpoint and produce a conflict map.
      p4migrate -c cmap checkpoint.1234
      
  5. Edit the produced conflict map and change case used in the right hand side of the mappings only to specify the case you would like to use to resolve each conflict
  6. Re-run p4migrate supplying the edited conflict map as the input translation map. This time, you should also request that an output checkpoint be written as if everything has been done correctly, it will be case-consistent.
      p4migrate -t cmap -c cmap2 -o checkpoint.1234.edited checkpoint.1234
      

    If all has gone well, then your conflicts map will be identical to your translation map and p4migrate will tell you that your checkpoint appears to be case-consistent.

    If your checkpoint still contains inconsistencies at this point, then review the differences between the translation map (cmap) and the conflicts map (cmap2) to determine what has gone wrong. If they are identical, please contact Perforce Technical Support for assistance.

  7. Copy your archive files to their new location. Note how you've changed the Map: in the pre-migration tasks, as to whether these should be in directory relative to the P4ROOT directory or in a new absolute path location.
  8. Re-run p4migrate for a third time to rename the archive files to match your case-consistent checkpoint. If the depot Map: uses relative paths and also has server.depot.root configured you may need to pass the serverId "-S $SERVERID", instead of the p4root "-r $P4ROOT".
      p4migrate -r $P4ROOT -R checkpoint.1234.edited
      
  9. Now recover from the edited checkpoint, start your Perforce server and run a "p4 verify -q //..." to check that the migration has succeeded. At this stage you should ignore any BAD! warnings produced by 'p4 verify'. The ones that require your immediate attention are the MISSING! messages. If in doubt, contact Perforce Technical Support for assistance.
  10. Assuming the previous step was OK, you now need to ensure that all your RCS-style archive files (*,v) have Unix-style line-endings rather than their original Windows-style line-endings. You can use the following to change the line endings:
      find $P4ROOT -type f \( -name "*,v" -o -name ".*,v" \) -print \
        -exec perl -p -i -e 's/\r\n/\n/' \{} \;
        
    The command syntax might be different depending on the specific platform and shell that you are using, but the above command should work in the majority of cases.

    NEVER RUN THIS STEP MORE THAN ONCE AGAINST THE SAME ARCHIVE FILES. If in doubt, contact Perforce Technical Support.

  11. Now run a 'p4 verify -q //...' again, and compare the output with that of the original 'p4 verify' you ran on Windows. Examine all the differences carefully.

    Note that "p4 verify" often reports ktext (or text+k) files as being "BAD" after such migrations. This is because keywords like $Id$ can expand differently if the case of the filename has changed and so cause the MD5 checksum of the revision to change. In these cases only you should use "p4 verify -v" to update the stored MD5 sum to reflect the new value. In all other cases, contact Perforce Technical Support.

Post Migration Tasks

These are items that you may want to consider doing after the migration.

Get users to re-sync files

As p4migrate may have changed the case of the files in your Server, it's recommneded that user sync their workspace to #none and then again to #head if they believe they have files that match the changed case paths. Note, they should also remove any intermediate directory paths and make sure these are consistent with the case changes. Failing to do this may cause users to be confused as to what files they currently have open.

Rebuild the haveMaps

The migration process corrects the case of the depotFile field in all relevant tables. In the case of tables that relate clientFile and depotFile together (db.locks, db.working, db.have), this can introduce a situation that requires a haveMap entry to be created. P4Migrate does not create this entry. To rebuild the haveMaps for 2007.3 servers onwards, you will need to run the following server command after the server migration.

    p4d -r $P4ROOT -xU BuildHaveMaps
    

For a large instance this may take some time to run.

Check Case Triggers

If you are moving to Unix and will still have active Windows users, you may want to consider adding a Change-submit trigger to check the case of the files being submitted. When using a case-sensitive server Windows users can accidentally submit files of names that already exist. Examples of such scripts to help prevent this can be found within the Perforce Workshop :

Case check triggers may have a performance impact on your system. We recommend acceptance testing your trigger in a test environment to better understand its performance characteristics.

Advanced Usage

Two-stage Parsing

It's possible to combine the second and third executions of p4-migrate into one to avoid a third scan of your checkpoint. If there are still case-inconsistencies in the checkpoint then the results will not be what you want, but a subsequent run with a case-consistent checkpoint should correct any problems.

To do this simply add the "-r" and "-R" flags to the command line. i.e.

  p4migrate -r $P4ROOT -R -t cmap -c cmap2 -o checkpoint.1234.edited checkpoint.1234
  

Generating a Journal Patch

As an alternative to creating a replacement checkpoint, p4migrate can generate a journal file that deletes offending records and inserts case-consistent replacement records. Simply add the -j option to the command line to use this feature.

  p4migrate -t cmap -c cmap2 -j -o journal.patch checkpoint.1234
  

Your mileage may vary when using this option because whilst the journal file is likely (but not guaranteed) to be smaller than the checkpoint, it can take longer than a checkpoint to apply.

Migrating from Unix to Windows

Whilst uncommon, migrating in this direction is not unheard of and p4migrate can be of some assistance.

When migrating from Unix to Windows, your meta-data is already consistent in its use of case so you don't have to worry about that. What you do have to be concerned about is filename collisions that could occur when you migrate to a Windows server. The classic example is two separate files in a Unix Perforce database: "makefile" and "Makefile". On Unix they are different files, but if you migrate to Windows they will immediately be treated as the same file. Even worse, their archive filenames will conflict and one will overwrite the other when you copy the files from Unix to Windows.

This problem also exists at the directory level so it may be manifest at any level in your repository. With directory name collisions, the directories concerned will effectively be merged into one which may in some cases be acceptable, but in others it will cause problems. Filename collisions are more severe.

p4migrate can help by identifying these collisions. It cannot help you to resolve them though. To identify the problems run p4migrate as if migrating from Windows-to-Unix and the contents of the conflict map will tell you where your collisions are.

  p4migrate -c collisions checkpoint.1234
  

For every mapping in the output conflict map, there is a filename or directory name collision. For example, a conflict mapping like this:

  //depot/Main/... //depot/Main/...
  

represents a directory level conflict whilst a mapping like this:

  //depot/main/www/Index.html //depot/main/www/Index.html
  

represents a file level conflict. If both conflicts are present, then your conflict map will look something like this:

  //depot/Main/...            //depot/Main/...
  //depot/Main/www/Index.html //depot/Main/www/Index.html
  

Having identified the collisions, you must now resolve them manually by renaming the files/directories as appropriate. This will require you to edit your checkpoint and also to rename the files on disk.

Archive File Conversion Issues

Text files have two different styles of line-endings (three if you count Macintosh) and the actual Operating System itself interprets files according to a specific convention. You must take this into consideration when you copy the archive files across to your new platform. For information about line-ending conversions and simple ways to do wholesale conversions of your archive files, please visit the following KB article for details: Cross-Platform Perforce Server Migration.

Basically, either take your LFs and convert them to CRLF (for Unix to Windows,) or take your CRLFs and convert them to LFs (for Windows to Unix.)

Contacting Support

Migrating a server from one OS to another is a complicated procedure. It is expected that Perforce Technical Support will be contacted to advise during this conversion. The easiest way to do this is to email support support-helix-core@perforce.com or contact Helix core support using the main Perforce Technical Support page.