The current version of p4migrate is 2024.1. This supports all servers up to and including 2024.1.
Using a Perforce Server greater than 2024.1 with this version of p4migrate may mean you see errors
in the conversion and is not supported
p4migrate is a tool that works with a checkpoint of your Perforce database in order to facilitate migrating your Perforce server between platforms.
p4migrate will help you generate a case-consistent checkpoint, by default this checkpoint will be marked as case sensitive. The p4migrate -C1 option allows you to generate a case insensitive checkpoint instead.
Windows and OSX are case-insensitive, Unix is case-sensitive. Similarly, Perforce on Windows and OSX is case-insensitive and Perforce on Unix is case-sensitive. This is because Perforce relies heavily on the semantics of the underlying filesystem on which your data is stored.
As Perforce on Windows is case-insensitive, the Perforce meta-data on a Windows server may contain references to files whose names differ only by case. On Windows these are references to the same file, however if you migrate meta-data in this state to Unix, they will be seen as references to different files.
This situation arises because Windows users are not always consistent in their use of case when referring to files and is called a case inconsistency.
As Unix is case sensitive, Unix users have little choice but to be consistent in their use of case if they want their commands to work so the same problem does not arise on the Unix platforms.
On Windows servers more recent than 2000.2, the Perforce server consistently uses lowercase names for the files and directories that make up the revision archives. This is not the meta-data (which is in the db.* files), but the depot directories in which the physical copies of your file revisions reside.
When you migrate to Unix, these archive files/directories must be renamed to match the case in your meta-data or the Perforce server will be unable to locate any files or revisions.
p4migrate solves both of the problems above. It first helps you to produce a case-consistent copy of your meta-data by editing a checkpoint according to rules you define; and then it can rename the archive files/directories to match the case in this edited checkpoint.
Note that it's not perfect. There are some types of case-inconsistency that are not possible to detect computationally as they may be intentional. It's unlikely, but they may. These p4migrate leaves for the user to resolve after the migration is complete. They all have very low impact and are easy to resolve.
So when you move a Perforce installation from Windows to Unix, there are essentially three steps to the process:
Before getting started with p4migrate, some definitions are useful
| Term | Explanation |
|---|---|
| conflict map | A mapping (in the Perforce sense) that describes the locations of case-inconsistency within your meta-data. The conflict map is produced by p4migrate from an analysis of your checkpoint. The conflict map contains one mapping for each inconsistency found. |
| translation map | A mapping (in the Perforce sense) supplied as input to p4migrate. All paths in your checkpoint are translated through this mapping before undergoing conflict analysis or being written to the output checkpoint. Typically an edited conflict map is used as the translation map. |
It's worth looking at a couple of example conflict/translation maps in more detail. Remember a conflict map is what p4migrate generates, a translation map is a conflict map after you've edited it to express your preferences.
So, if p4migrate detects that you have been inconsistent in your use of case when referring to files in the //depot/main/www/... directory, it will generate a conflict map entry to help you resolve this conflict.
The left hand side of the mapping is in mixed case and is the source pattern
The right hand side of the mapping is in mixed case and is the destination pattern.
The mappings are case-insensitive on all platforms.
Initially both sides are identical and are set to the most commonly used variant of the path with conflicts. You change the right hand side only.
So in the example above, the conflict mapping produced by p4migrate might look like this:
//depot/main/WWW/... //depot/main/WWW/...
So in this case, p4migrate offers you the //depot/main/WWW/... variant as your default because that is the most frequently observed variant in the checkpoint file. If that's what you want, then you're ready to proceed, but if not then you should edit the file and change the case (and only the case) to suit your preferences. Then the file is ready to be used as a translation map.
What if you'd been inconsistent in your use of the "main" directory too? Well in that case the mappings might have looked like this:
//depot/Main/... //depot/Main/...
//depot/Main/WWW/... //depot/Main/WWW/...
So in this case, if you're unhappy with the case of the "Main" directory, you must change it in the right hand side of both mapping lines.
The mappings in p4migrate conflict/translation maps are just plain old Perforce mappings. This means that there are two golden rules:
And some p4migrate specific rules
In practical terms this means that you must
or p4migrate will not work as expected. The second rule is tough to understand so an example is in order. Say p4migrate generates a mapping like this:
//depot/QA/Gastar/... //depot/QA/Gastar/...
and you edit it before using it as a translation map and you alter the case of the QA directory as well as changing the case of the leaf-directory (Gastar) where the conflict actually occurred.
//depot/QA/Gastar/... //depot/qa/gastar/...
In this case, you must also add a new mapping for the QA directory to your translation map or you will have introduced a new conflict all by yourself for //depot/QA/... paths other than the Gastar subdirectory.
//depot/QA/... //depot/qa/... //depot/QA/Gastar/... //depot/qa/gastar/...
Note that p4migrate just appends new conflict mappings to the existing translation map so in some circumstances p4migrate itself may generate mappings that are not in the correct order. Therefore you should always review your mappings carefully.
//depot/production/... //depot/production/... //depot/production/gastar/... //depot/production/gastar/... //depot/qa/... //depot/qa/... //depot/qa/gastar/... //depot/qa/gastar/...
//depot/production/gastar/... //depot/production/gastar/... //depot/qa/gastar/... //depot/qa/gastar/... //depot/production/... //depot/production/... //depot/qa/... //depot/qa/...
p4migrate has 3 locale based configurations, configured by the -L locale setting
If the checkpoint is from a unicode mode server then utf8 specific case folding will be used.
if no -L locale option is passed, the 'C' locale will be used by default
There is a chance, that for a given locale setting, p4migrate will miss some case folding conversions. If this happens, multiple permutations of a given case insensitive path may still exist in the resulting checkpoint. You may wish to try the migration again using a different locale setting to see if it works better.
Policies can be used to apply a global transformation to all paths in your repository. If it's your policy that all filenames should be lower- or upper-case, then p4migrate can correct all exceptions to this policy and bring all your pathnames into line.
p4migrate -p upper -c conflicts -o checkpoint.new checkpoint.NNNN p4migrate -p lower -c conflicts -o checkpoint.new checkpoint.NNNN
There's also a policy called 'head' which will not transform the paths in any way, but rather affects the path that will be chosen for inclusion in the conflict map. Instead of choosing the most frequently-used case-variant, it will select the case used by the head revision (in the case of conflicts) at the filename level and, for conflicts at the folder level, it will choose the most recently encountered variant used by the head revisions of the files underneath that folder.
p4migrate -p head -c conflicts -o checkpoint.new checkpoint.NNNN
There are some caveats with the use of policies:
You can override your policy using the Translation Map (-t transmap) to specify the exceptions to the rule as the Translation Map is applied after the policy. When you do this, ensure that you provide a complete set of mappings and use the -c flag with p4migrate so that any conflicts (perhaps caused by missing mappings) are recorded in a conflicts map so that you can correct your translation map and try again. For example:
p4migrate -p upper -c conflicts -t transmap -o checkpoint.new checkpoint.NNNN
Note, using this feature you will need to confirm that you have understood that this can casue unexpected results if applied incorrectly.
The "Translation Map" is in the same format of the conflict maps. Any path that is mentioned in the the Translation Map will be applied after the policy. In the above the policy is "upper". If the Translation Map contains:
//depot/production/... //DEPOT/production/...
//depot/production/gastar/... //DEPOT/production/gastar/...
The the paths starting with "//DEPOT/production/..." and "//DEPOT/production/gastar/..." will all be in lower case matching the translation map, and all other depot will be in upper case Matching the policy, upper. For example, your final depot layout would appear as:
//DEPOT/FOO.TXT
//DEPOT/DEV/FOO.TXT
//DEPOT/production/LIVE/FOO.TXT
//DEPOT/production/gastar/QA/FOO.TXT
Applying policies and Translation Maps at the same time can lead to unexpected results if applied incorectly. If in doubt, contact Perforce Technical Support for assistance.
Usage: p4migrate [options] <checkpoint file>
where options are:
-c <filename> - Filename to use for the output translation map,
also known as the 'conflict map'.
Conflicts in your checkpoint are recorded in this
file by generated mappings from the lowercase
variant of the filename to the most frequently
observed variant in your checkpoint. You should
edit this file and change the mappings to meet
your requirements and then supply the edited file
as the input translation map using the -t flag.
-C0 - Generate a case sensitive checkpoint (default)
-C1 - Generate a case insensitive checkpoint
-j - Produce the output as a journal patch rather
than a replacement checkpoint.
-l <filename> - Filename to use for logfile. Defaults to stderr.
-L <locale> - The locale to use during the migration.
Not used if the source checkpoint is from a 'unicode' server.
The 'C' locale is set by default, and is the fastest for migrating,
but will not support case folding of multi-byte characters.
Any other locale may be specified, but it may slow down processing,
and will require that the specified locale is installed on the machine.
If the locale specified contains the string 'UTF-8' then utf8
case folding will be enabled.
-o <filename> - Filename to use for the output checkpoint.
After mapping records through the input translation
map, the edited records are written out to this
file. If no further conflicts remain, then this is
a case-consistent checkpoint. If conflicts remain
unresolved then this checkpoint should not be used
rebuild your database. You may pass '-' to use
stdout.
-p <policy> - Name of a built-in policy to apply. Valid policies
are:
lower - convert all paths to lowercase
upper - convert all paths to uppercase
head - convert all paths to the variant used
by the head revision
Note: The values for upper and lowercase characters
are very dependent on your locale. If your
checkpoint contains characters in different
character sets you should exercise great care with
the use of this option.
In particular, you should run 'p4 verify -q //...'
and check the output very carefully.
Note 2: Policies are applied before the translation
map, so you may override the policy for specific
paths using the translation map. You will likely
need several mappings to get this right. e.g.
with a policy of 'upper' you might use something
like this:
//depot/... //Depot/...
//depot/main/... //Depot/Main/...
//depot/main/www/... //Depot/Main/www/...
i.e. a mapping for every part of the tree that
you want to exempt from the policy
-R - Attempt to rename the archive files to match the
case in the translated checkpoint. This option
is only available on case-sensitive platforms.
-r <p4root> - Your P4ROOT directory, this option may be needed if -R is used,
and the server.depot.root configurable is not found.
-s - Will convert the input checkpoint to
'standard checkpoint' format.
-S <serverId> - The ServerId for this server, this may be required when
server.depot.root configurable is found.
-t <filename> - Filename for the input translation map.
The paths in your checkpoint are remapped through
the mappings in this file to resolve conflicts
due to inconsistent use of case. Default: none
See also -c.
-u - Source checkpoint is from unicode mode server.
This option is only needed if the checkpoint is missing a
Checkpoint header journal note.
-v <level> - Output verbosity, (0-6) higher levels provide more detail
-V - Print version details and exit.
For instance the 64 bit linux release of r20.2 can be found here: http://ftp.perforce.com/perforce/r20.2/bin.linux26x86_64/p4migrate
Unlike other support tools, p4migrate works with any previous release of perforce, you don't need to exactly match your p4d version
Often when diffing checkpoints there are a many differences because of optional quoting around integer values. The -s option will convert the input checkpoint into a 'standard checkpoint' format to allow textual diffs to show content changes cleanly.
p4migrate -s -o checkpointOut checkpointIn
will read checkpointIn and output it to checkpointOut in standard checkpoint format
If the -R flag is used, p4migrate will attempt to rename the archive files to match the given checkpoint. It uses the following algorihm to find the archive files base directory:
Group names used in properties and permissions are renamed to match the case of existing db.group and db.groupx table entries.
Archive depots names will automatically be converted to lower case. This is to maintain consistency with the rev records. If you have files that have been archived using the "p4 archive" command, the depot path for the archived files will be consistent with the translation map. For example, if your translation map is as follows:
//depot/production/spares/... //depot/Production/Spares/...And your archive Depot is called
BUCKET, the migrated paths will become:
//bucket/depot/Production/Spares/...
When renaming the archive files as described below, it is advised that you have your archive depot attached to the P4ROOT.
Failing to do so will mean that P4Migrate will not be able to rename any archived files and you will need to do so manually before restoring the archive files.
Stream Depots and Streams
Stream Depots and Streams are treated in the same way as other file names within P4Migrate. The output within the cmap file may appear similar to:
//STRDepot/Prod_650P_DEV //STRDepot/Prod_650P_DEV
//STRDepot/Prod_650P_DEV/... //STRDepot/Prod_650P_DEV/...
With //STRDepot/Prod_650P_DEV as the name of the Stream and //STRDepot/Prod_650P_DEV/... the files underneath the Stream.
The same rules that apply to non-stream file paths apply to both the Stream name and path.
Unloaded Data
Currently p4migrate does not support unloaded data as this contains meta data which is written to disk in the archive location which may not be available to p4migrate.
This metadata may contain case inconsistent data and as such should be reloaded to make sure that all any case issues in this data are resolved. Before you create your checkpoint to test, check the output of:
p4 labels -U
p4 clients -U
p4 streams -U
Migrating from Windows to Unix
This is the most common migration and the one which p4migrate was primarily designed to support; to move from a case-insensitive platform and run a case-sensitive Perforce Server on a Unix based platform.
If you are wanting to run a case-insensitive Perforce Server on a Unix based platform, then you should use the p4d "-C1" flag as p4migrate is not required for
this purpose. Please contact Perforce Support for further information on this flag.
Pre-Migration tasks
- Make sure that the depot Map: paths are either relative to P4ROOT or server.depot.root (if configured) or changed to the new absolute location for the target platform.
If the Map: needs to be changed, you should do so after starting the Perforce Server on localhost. This can be done as follows:
-
Shutdown the Perforce Server by running:
p4 admin stop
-
Restart the Perforce Server listening only on localhost:1666
p4d -r P4ROOT -p localhost:1666
-
Connect to the Perforce Server specifying the localhost:1666 port and change the depot Map:
p4 -p localhost:1666 login
p4 -p localhost:1666 depot depot
-
Change depot Map: be relative or be the full path for the location of the archive files on your Perforce Server. For example, a relative depot Map: path:
Map: depot/...
and an absolute path:
Map: /perforce/archives/depot/...
-
Repeat this for all depots on your Perforce Server
-
Shutdown the Perforce Server
p4 -p localhost:1666 admin stop
Migration tasks
-
Take a fresh checkpoint and a full backup. Do not be tempted to skip this step!
-
Run a "p4 verify -q //..." to ensure the integrity of your archive files.
-
Copy the checkpoint created, the archive files and license file to your target machine.
-
-
Download p4migrate for your target machine from the same place you'd look for 'p4d', in:
"ftp://ftp.perforce.com/perforce/RELEASE/ARCHITECTURE/p4migrate/".
-
For instance the 64 bit linux release of r22.1 is here:
ftp://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4migrate.
- unlike other support tools, p4migrate works with any previous release of perforce, you don't need to exactly match your p4d version
-
Run p4migrate to identify the case-conflicts within your checkpoint and produce
a conflict map.
p4migrate -c cmap checkpoint.1234
-
Edit the produced conflict map and change case used in the right hand side of
the mappings only to specify the case you would like to use to resolve each
conflict
-
Re-run p4migrate supplying the edited conflict map as the input translation map.
This time, you should also request that an output checkpoint be written as if
everything has been done correctly, it will be case-consistent.
p4migrate -t cmap -c cmap2 -o checkpoint.1234.edited checkpoint.1234
If all has gone well, then your conflicts map will be identical to your
translation map and p4migrate will tell you that your checkpoint appears to
be case-consistent.
If your checkpoint still contains inconsistencies at this point, then review the
differences between the translation map (cmap) and the conflicts map (cmap2)
to determine what has gone wrong. If they are identical, please contact Perforce
Technical Support for assistance.
-
Copy your archive files to their new location. Note how you've changed the Map: in the
pre-migration tasks, whether these should be in directory relative to the P4ROOT directory
or in a new absolute path location.
-
Re-run p4migrate for a third time to rename the archive files to match your
case-consistent checkpoint. If the depot Map: uses relative paths and also has server.depot.root configured
you may need to pass the serverId "-S $SERVERID", instead of the p4root "-r $P4ROOT".
p4migrate -r $P4ROOT -R checkpoint.1234.edited
-
Now recover from the edited checkpoint, start your Perforce server and run a
"p4 verify -q //..." to check that the migration has succeeded. At this stage
you should ignore any BAD! warnings produced by 'p4 verify'.
The ones that require your immediate attention are the MISSING!
messages. If in doubt, contact Perforce Technical Support for assistance.
-
Assuming the previous step was OK, you now need to ensure that all your
RCS-style archive files (*,v) have Unix-style line-endings rather than
their original Windows-style line-endings. You can use the following
to change the line endings:
find $P4ROOT -type f \( -name "*,v" -o -name ".*,v" \) -print \
-exec perl -p -i -e 's/\r\n/\n/' \{} \;
The command syntax might be different depending on the specific platform and
shell that you are using, but the above command should work in the majority of
cases.
NEVER RUN THIS STEP MORE THAN ONCE AGAINST THE SAME ARCHIVE FILES.
If in doubt, contact Perforce Technical Support.
-
Now run a 'p4 verify -q //...' again, and compare the output with that of
the original 'p4 verify' you ran on Windows. Examine all the differences
carefully.
Note that "p4 verify" often reports ktext (or text+k) files as being "BAD"
after such migrations. This is because keywords like $Id$ can expand
differently if the case of the filename has changed and so cause the MD5
checksum of the revision to change. In these cases only
you should use "p4 verify -v" to update the stored MD5 sum to reflect the
new value. In all other cases, contact Perforce Technical Support.
Post Migration Tasks
These are items that you may want to consider doing after the migration.
Get users to re-sync files
As p4migrate may have changed the case of the files in your Server, it's recommneded
that user sync their workspace to #none and then again to #head if they believe
they have files that match the changed case paths. Note, they should also remove
any intermediate directory paths and make sure these are consistent with the case
changes. Failing to do this may cause users to be confused as to what files they currently
have open.
Rebuild the haveMaps
The migration process corrects the case of the depotFile field in all
relevant tables. In the case of tables that relate clientFile and depotFile
together (db.locks, db.working, db.have), this can introduce a situation that
requires a haveMap entry to be created. P4Migrate does not create this entry.
To rebuild the haveMaps for 2007.3 servers onwards, you will need to run the
following server command after the server migration.
p4d -r $P4ROOT -xU BuildHaveMaps
For a large instance this may take some time to run.
Check Case Triggers
If you are moving to Unix and will still have active Windows users, you may want
to consider adding a Change-submit trigger
to check the case of the files being submitted. When using a case-sensitive server
Windows users can accidentally submit files of names that already exist. Examples
of such scripts to help prevent this can be found within the
Perforce Workshop :
-
CheckCaseTrigger.py,
from Sven Erik Knop's
P4Python Library.
-
Checkcase.rb,
from Tony Smith's
P4Ruby Library.
Case check triggers may have a performance impact on your system. We recommend
acceptance testing your trigger in a test environment to better understand its
performance characteristics.
Advanced Usage
Two-stage Parsing
It's possible to combine the second and third executions of p4-migrate into one
to avoid a third scan of your checkpoint. If there are still case-inconsistencies in
the checkpoint then the results will not be what you want, but a subsequent run
with a case-consistent checkpoint should correct any problems.
To do this simply add the "-r" and "-R" flags to the command line. i.e.
p4migrate -r $P4ROOT -R -t cmap -c cmap2 -o checkpoint.1234.edited checkpoint.1234
Generating a Journal Patch
As an alternative to creating a replacement checkpoint, p4migrate can generate a
journal file that deletes offending records and inserts case-consistent replacement
records. Simply add the -j option to the command line to use this
feature.
p4migrate -t cmap -c cmap2 -j -o journal.patch checkpoint.1234
Your mileage may vary when using this option because whilst the journal file is
likely (but not guaranteed) to be smaller than the checkpoint, it can take longer
than a checkpoint to apply.
Migrating from Unix to Windows
Whilst uncommon, migrating in this direction is not unheard of and
p4migrate can be of some assistance.
When migrating from Unix to Windows, your meta-data is already consistent
in its use of case so you don't have to worry about that. What you do
have to be concerned about is filename collisions that could occur when
you migrate to a Windows server. The classic example is two separate files
in a Unix Perforce database: "makefile" and "Makefile". On Unix they are
different files, but if you migrate to Windows they will immediately be
treated as the same file. Even worse, their archive filenames will conflict
and one will overwrite the other when you copy the files from Unix to
Windows.
This problem also exists at the directory level so it may be manifest at
any level in your repository. With directory name collisions, the
directories concerned will effectively be merged into one which may in
some cases be acceptable, but in others it will cause problems. Filename
collisions are more severe.
p4migrate can help by identifying these collisions. It cannot help you to
resolve them though. To identify the problems run p4migrate as if migrating
from Windows-to-Unix and the contents of the conflict map will tell you where
your collisions are.
p4migrate -c collisions checkpoint.1234
For every mapping in the output conflict map, there is a filename or
directory name collision. For example, a conflict mapping like this:
//depot/Main/... //depot/Main/...
represents a directory level conflict whilst a mapping like this:
//depot/main/www/Index.html //depot/main/www/Index.html
represents a file level conflict. If both conflicts are present, then
your conflict map will look something like this:
//depot/Main/... //depot/Main/...
//depot/Main/www/Index.html //depot/Main/www/Index.html
Having identified the collisions, you must now resolve
them manually by renaming the files/directories as appropriate. This
will require you to edit your checkpoint and also to rename the files
on disk.
Archive File Conversion Issues
Text files have two different styles of line-endings (three if you count
Macintosh) and the actual Operating System itself interprets files
according to a specific convention. You must take this into consideration
when you copy the archive files across to your new platform. For
information about line-ending conversions and simple ways to do wholesale
conversions of your archive files, please visit the following KB article for details: Cross-Platform Perforce Server Migration.
Basically, either take your LFs and convert them to CRLF (for Unix to
Windows,) or take your CRLFs and convert them to LFs (for Windows to
Unix.)
Contacting Support
Migrating a server from one OS to another is a complicated procedure. It
is expected that Perforce Technical Support will be contacted to advise
during this conversion. The easiest way to do this is to email support
support-helix-core@perforce.com or
contact Helix core support using the main
Perforce Technical Support page.