Introduction to Federated Services

Perforce federated architecture aims to make simple tasks easy and complex tasks possible; it allows you to start simply and to grow incrementally in response to the evolving needs of your business.

This chapter describes the different types of Perforce servers and explains how you combine these to solve usability and performance issues. In addition, this chapter discusses the issues that affect service federation independently of the particular type of service used, issues like user authentication and communication among federated services. Subsequent chapters in this book describe each type of service in detail.

To make best use of the material presented in this book, you should be familiar with the Helix Versioning Engine Administrator Guide: Fundamentals.

Overview

Helix Versioning Engine Administrator Guide: Fundamentals explains how you create, configure, and maintain a single Perforce Server. For most situations, a single server that is accessible to all users can take care of their needs with no problems. However, as business grows and usage expands, you might find yourself needing to deploy a more powerful server-side infrastructure. You can do so using three different types of Perforce services:

  • Proxy

    Where bandwidth to remote sites is limited, you can use a Perforce proxy to improve performance by mediating between Perforce clients and the versioning service. Proxies cache frequently transmitted file revisions. By intercepting requests for cached revisions, the proxy reduces demand on the server and keeps network traffic to a minimum.

    The work needed to install and configure a proxy is minimal: the administrator needs to configure a proxy on the side of the network close to the users, configure the users to access the service through the proxy, and then configure the proxy to access the Perforce versioning service. You do not need to backup the proxy cache directory: in case of failure, the proxy can reconstruct the cache based on the Perforce server metadata. For complete information about using proxies, see “Perforce Proxy”.

  • Broker

    A Perforce broker mediates between clients and server processes (including proxies) to implement policies in your federated environment. Such policies might direct specific commands to specific servers or they might restrict the commands that can be executed. You can use a broker to solve load-balancing, security, or other issues that can be resolved by sorting requests directed to one or more Perforce servers.

    The work needed to install and configure a broker is minimal: the administrator needs to configure the broker and configure the users to access the Perforce server through the broker. Broker configuration involves the use of a configuration file that contains rules for specifying which commands individual users can execute and how commands are to be redirected to the appropriate Perforce service. You do not need to backup the broker. In case of failure, you just need to restart it and make sure that its configuration file has not been corrupted. For complete information about using the broker, see “The Perforce Broker”.

  • Replica

    A replica duplicates server data; it is one of the most versatile elements in a federated architecture. You can use it to provide a warm standby server, to reduce load and downtime on a primary server, to support build farms, or to forward requests to a central server. This latter use case, of the forwarding replica, can be implemented using a commit-edge architecture, which improves performance and addresses the problem of remote access.

    The amount of administrative work needed for installing, configuring, and managing replicates varies with the type of replicate used. For information about the handling of different replicate types, see “Perforce Replication”. For information about commit-edge deployments, see “Commit-edge Architecture”.

In addition to these three types of servers, to simplify administrative work, a federated architecture might also include servers dedicated to centralized authorization and changelist numbering. For more information, see Configuring centralized authorization and changelist servers. The next section explains how you might combine these types to address various user needs.

User scenarios

Which types of servers you use and how you combine them varies with your needs. The following discussion examines what servers you’d choose to support high availability, geographical distribution, build automation, scalability, and governance.

Availability

As users become more dependent on a Perforce server, you might want to minimize server downtime. By deploying additional replicas and brokers, you can set up online checkpoints so that users can continue work while you make regular backups. This more sophisticated infrastructure also allows you to perform regular maintenance using p4 verify or p4 dbverify without server downtime. You can re-route requests targeted for one machine to another during routine system maintenance or operating system upgrades.

Should your primary server fail, this server infrastructure allows you to fail over to a standby machine with minimal downtime. If the backup server is deployed to a different machine, this architecture can also be used for disaster recovery. Replica types best suited for failover and disaster recovery are read-only replicas and forwarding replicas.

Remote offices

As your organization grows and users are added in remote offices, you need to provide remote users with optimal performance when they access the Perforce server.

The primary challenge to performance in a geographically distributed environment is network latency and bandwidth limitations. You can address both issues using Perforce proxies and replicas. These reduce network traffic by caching file content and metadata in the remote office and by servicing requests locally whenever possible. Up to 98% of user requests can be handled by the servers in the remote office.

You can also configure brokers to re-route requests in cases of outage and to assist in managing off-hour requests that occur with a global workforce.

Build/test automation

With the increasing use of build and test automation, it is important to deploy a server architecture that addresses the growing workload. Automated build and test tools can impose a massive workload on the storage and networking subsystems of your Perforce server. This workload includes agile processes and continuous delivery workflows, reporting tools that run frequent complex queries, project management tools that need close integration with server data, code search tools, static analysis tools, and release engineering tools that perform automated branch integrations and merges.

To improve user experience, you need to shift this growing workload to dedicated replica servers and relieve your master server of those tasks, enabling it to concentrate on servicing interactive user requests.

Scalability

As your organization grows, you need to grow your infrastructure to meet its needs.

  • The use of advanced software engineering tools will benefit from having additional server-side resources. Deploying Perforce proxies, replicas, and brokers allows you to add additional hardware resources and enables you to target each class of request to an appropriately-provisioned server, using your highest-performance infrastructure for your most critical workloads while redirecting lower-priority work to spare or under-utilized machines.
  • As the number of users and offices grows you can plan ahead by provisioning additional equipment. You can deploy Perforce proxies, replicas, and brokers to spread your workload across multiple machines, offload network and storage usage from your primary data center to alternate data centers, and support automation workloads by being able to add capacity.

Policy-based governance

As usage grows in size and sophistication, you might want to establish and maintain policies and procedures that lead to good governance.

For example, you might want to use repository naming conventions to make sure that your repository remains well organized and easy to navigate. In addition you might find that custom workflows, such as a change review process or a release delivery process, are best supported by close integration with your version control infrastructure.

You can use brokers in your federated deployment to filter requests, enforce policy, and re-route commands to alternate destinations. This provides a powerful infrastructure for enforcing your organization’s policies. Deploying trigger scripts in your servers instances enables additional integration with other software development and management tools.

Setting up federated services

This section describes some of the issues that administration must address in setting up a federated environment.

General guidelines

Following these guidelines will simplify the administration and maintenance of federated environments:

  • Every server should be assigned a server ID; it is best if the serverID is the same as the server name. Use the p4 server command to identify each server in your network.
  • Every server should have an assigned (and preferably unique) service user name. This simplifies the reading of logs and provides authentication and audit trails for inter-server communication. Assign service users strong passwords. Use the p4 server command to assign a service user name.
  • Enable structured logging on all your services. Doing so can greatly simplify debugging and analysis, and is also required in order to use the p4 journaldbchecksums command to verify the integrity of a replica.
  • Configure each server to reject operations that reduce its disk space below the limits defined by that service’s filesys.*.min configurables.
  • Monitor the integrity of your replicas by using the integrity.csv structured server log and the p4 journaldbchecksums command. See Verifying replica integrity for details.

Authenticating users

Users must have a ticket for each server they access in a federated environment. The best way to handle this requirement is to set up a single login to the master, which is then valid across all replica instances. This is particularly useful with failover configurations, when you would otherwise have to re-login to the new master server.

You can set up single-sign-on authentication using two configurables:

  • Set auth.id to the same value for all servers participating in a distributed configuration.
  • Enable rpl.forward.login (set to 1) for each replica participating in a distributed configuration.

There might be a slight lag while you wait for each instance to replicate the db.user record from the target server.

Connecting services

Services working together in a federated environment must be able to authenticate and trust one another.

  • When using SSL to securely link servers, brokers, and proxies together, each link in the chain must trust the upstream link.
  • It is best practice (and mandatory at security level 4) to use ticket-based authentication instead of password-based authentication. This means that each service user for each server in the chain must also have a valid login ticket for the upstream link in the chain.

Managing trust between services

The user that owns the server, broker, or proxy process is typically a service user. As the administrator, you must create a P4TRUST file on behalf of the service user by using the p4 trust command) that recognizes the fingerprint of the upstream Perforce service.

By default, a user’s P4TRUST file resides in their home directory as .p4trust. Ensure that the P4TRUST variable is correctly set so that when the user (often a script or other automation tool) that actually invokes the p4d, p4p, or p4broker executable is able to read filename to which P4TRUST points in the invoking user’s environment.

Further information is available in the Helix Versioning Engine Administrator Guide: Fundamentals.

Managing tickets between services

When linking servers, brokers, and proxies together, each service user must be a valid service user at the upstream link, and it must be able to authenticate with a valid login ticket. Follow these steps to set up service authentication:

  1. On the upstream server, use p4 user to create a user of type service, and p4 group to assign it to a group that has a long or unlimited timeout.

    Use p4 passwd to assign the service user a strong password.

  2. On the downstream server, use p4 login to log in to the master server as the newly-created service user, and to create a login ticket for the service user that exists on the downstream server.
  3. Ensure that the P4TICKET variable is correctly set when the user (often a script or other automation tool) that actually invokes the downstream service, does so, so that the downstream service can correctly read the ticket file and authenticate itself as the service user to the upstream service.

Managing SSL key pairs

When configured to accept SSL connections, all server processes (p4d, p4p, p4broker), require a valid certificate and key pair on startup.

The process for creating a key pair is the same as it is for any other server: set P4SSLDIR to a valid directory with valid permissions, and use the following commands to generate pairs of privatekey.txt and certificate.txt files, and make a record of the key’s fingerprint.

  • Server: use p4d -Gc to create the key/certificate pair and p4d -Gf to display its fingerprint.
  • Broker: use p4broker -Gc to create the key/certificate pair and p4broker -Gf to display its fingerprint.
  • Proxy: use p4p -Gc to create the key/certificate pair and p4p -Gf to display its fingerprint.

You can also supply your own private key and certificate. Further information is available in the Helix Versioning Engine Administrator Guide: Fundamentals.

Backing up and upgrading services

Backing up and upgrading services in a federated environment involve special considerations. This section describes the issues that you must resolve in this environment.

Backing up services

How you backup federated services depends upon the service type:

  • Server

    Follow the backup procedures described in the Helix Versioning Engine Administrator Guide: Fundamentals. If you are using an edge-commit architecture, both the commit server and the edge servers must be backed up. Use the instructions given in Backup and high availability / disaster recovery (HA/DR) planning.

    Backup requirements for replicas that are not edge servers vary depending on your site’s requirements.

  • Broker: the broker stores no data locally; you must backup its configuration file manually.
  • Proxy: the proxy requires no backups and, if files are missing, automatically rebuilds its cache of data. The proxy contains no logic to detect when diskspace is running low. Periodically monitor your proxy to ensure it has sufficient diskspace for continued operation.

Upgrading services

Servers, brokers, and proxies must be at the same release level in a federated environment. When upgrading use a process like the following:

  1. Shut down the furthest-upstream service or commit server and permit the system to quiesce.
  2. Upgrade downstream services first, starting with the replica that is furthest downstream, working upstream towards the master or commit server.
  3. Keep downstream services stopped until the server immediately upstream has been upgraded.

Configuring centralized authorization and changelist servers

There are cases where rather than using federated services you want to use a collection of servers that have a shared user base. In this situation, you probably want to use specialized servers to simplify user authentication and to guarantee unique change list numbers across the organization. The following subsections explain how you create and use these servers: P4AUTH for centralized authentication and P4CHANGE to generate unique changelist numbers.

Centralized authorization server (P4AUTH)

If you are running multiple Perforce servers, you can configure them to retrieve protections and licensing data from a centralized authorization server. By using a centralized server, you are freed from the necessity of ensuring that all your servers contain the same users and protections entries.

Note

When using a centralized authentication server, all outer servers must be at the same (or newer) release level as the central server.

If a user does not exist on the central authorization server, that user does not appear to exist on the outer server. If a user exists on both the central authorization server and the outer server, the most permissive protections of the two lines of the protections table are assigned to the user.

You can use any existing Perforce Server in your organization as your central authorization server. The license file for the central authorization server must be valid, as it governs the number of licensed users that are permitted to exist on outer servers. To configure a Perforce Server to use a central authorization server, set P4AUTH before starting the server, or specify it on the command line when you start the server.

If your server is making use of a centralized authorization server, the following line will appear in the output of p4 info: 

...
Authorization Server: [protocol:]host:port

Where [protocol:]host:port refers to the protocol, host, and port number of the central authorization server. See Specifying hosts.

In the following example, an outer server (named server2) is configured to use a central authorization server (named central). The outer server listens for user requests on port 1999 and relies on the central server’s data for user, group, protection, review, and licensing information. It also joins the protection table from the server at central:1666 to its own protections table.

For example:

$ p4d -In server2 -a central:1666 -p 1999

Note

On Windows, configure the outer server with p4 set -S as follows: 

C:\> p4 set -S "Outer Server" P4NAME=server2
C:\> p4 set -S "Outer Server" P4AUTH=central:1666
C:\> p4 set -S "Outer Server" P4PORT=1999

When you configure a central authorization server, outer servers forward the following commands to the central server for processing:

Command Forwarded to auth server? Notes

p4 group

Yes

Local group data is derived from the central server.

p4 groups

Yes

Local group data is derived from the central server.

p4 license

Yes

License limits are derived from the central server. License updates are forwarded to the central server.

p4 passwd

Yes

Password settings are stored on, and must meet the security level requirements of, the central server.

p4 review

No

Service user (or remote) must have access to the central server.

p4 reviews

No

Service user (or remote) must have access to the central server.

p4 user

Yes

Local user data is derived from the central server.

p4 users

Yes

Local user data is derived from the central server.

p4 protect

No

The local server’s protections table is displayed if the user is authorized (as defined by the combined protection tables) to edit it.

p4 protects

Yes

Protections are derived from the central server’s protection table as appended to the outer server’s protection table.

p4 login

Yes

Command is forwarded to the central server for ticket generation.

p4 logout

Yes

Command is forwarded to the central server for ticket invalidation.

Limitations and notes

  • All servers that use P4AUTH must have the same Unicode setting as the central authorization server.

  • Setting P4AUTH by means of a p4 configure set P4AUTH=[protocol:]server:port command requires a restart of the outer server.

    If you need to set P4AUTH for a replica, use the following syntax:

    p4 configure set ServerName#P4AUTH=[protocol:]server:port

  • If you have set P4AUTH, no warning will be given if you delete a user who has an open file or client.

  • To ensure that p4 review and p4 reviews work correctly, you must enable remote depot access for the service user (or, if no service user is specified, for a user named remote) on the central server.

    Note: There is no remote type user; there is a special user named remote that is used to define protections for a remote depot.

  • To ensure that the authentication server correctly distinguishes forwarded commands from commands issued by trusted, directly-connected users, you must define any IP-based protection entries in the Perforce service by prepending the string “proxy-” to the [protocol:]host:port definition.
  • Protections for non-forwarded commands are enforced by the outer server and use the plain client IP address, even if the protections are derived from lines in the central server’s protections table.

Centralized changelist server (P4CHANGE)

By default, Perforce servers do not coordinate the numbering of changelists. Each Perforce Server numbers its changelists independently. If you are running multiple servers, you can configure your servers to refer to a centralized changelist server from which to obtain changelist numbers. Doing so ensures that changelist numbers are unique across your organization, regardless of the server to which they are submitted.

Note

When using a centralized changelist server, all outer servers must be at the same (or newer) release level as the central server.

To configure a Perforce Server to use a centralized changelist server, set P4CHANGE before starting the second server, or specify it on the p4d command line with the -g option: 

$ p4d -In server2 -g central:1666 -p 1999

Note

On Windows, configure the outer server with p4 set -S as follows: 

C:\> p4 set -S "Outer Server" P4NAME=server2
C:\> p4 set -S "Outer Server" P4CHANGE=central:1666
C:\> p4 set -S "Outer Server" P4PORT=1999

In this example, the outer server (named server2) is configured to use a centralized changelist server (named central). Whenever a user of the outer server must assign a changelist number (that is, when a user creates a pending changelist or submits one), the centralized server’s next available changelist number is used instead.

There is no limit on the number of servers that can refer to a centralized changelist server. This configuration has no effect on the output of the p4 changes command; p4 changes lists only changelists from the currently connected server, regardless of whether it generates its own changelist numbers or relies on a centralized changelist server.

If your server is making use of a centralized changelist server, the following line will appear in the output of p4 info: 

...
Changelist Server: [protocol:]host:port

Where [protocol:]host:port refers to the protocol, host, and port number of the centralized changelist server.

Verifying shelved files

The verification of shelved files lets you know whether your shelved archives have been lost or damaged.

If a shelf is local to a specific edge server, you must issue the p4 verify -S command on the edge server where the shelf was created. If the shelf was promoted, run the p4 verify -S on the commit server.

You may also run the p4 verify -S t command on a replica to request re-transfer of a shelved archive that is missing or bad. Re-transferring a shelved archive from the master only works for shelved archives that are present on the master; that is, for a shelf that was originally created on the master or that was promoted if it was created on an edge server.