Perforce Insights 2015.1 ======================== Perforce Insights is a tool that presents vital information about how your projects, codelines, and teams are performing. Insights draws on data from the Perforce Server (P4D) and logs, so you can quickly understand what's going on. Rather than querying your Perforce service directly, Insights exports metadata from Perforce into a PostgreSQL database. Initially, the database can be seeded with a checkpoint, but after that updates to Perforce can be streamed in real time into Insights. The web application Insights provides a web front end for accessing this data, which is rendered in graph form. Access control is via Perforce, and users can configure their own dashboards to control what they see and how it is displayed. There are two versions of this README - README.txt and README.html. Both contain the same instructions, so use whichever is most convenient for you. There are also two versions of the package that contains this README - a .zip file which is optimized for Microsoft Windows (text files within it have DOS style line endings) and a .tgz file which is optimized for UNIX and Linux (text files have UNIX style line endings). Either will work on any supported platform, but you might experience oddities with your text editor when editing UNIX text files on MS Windows and vice versa. There is also documentation and guides available online at http://www.perforce.com/perforce/r15.1/manuals/p4insights/ Alternative Packages -------------------- There are also Linux packages available for Insights (available as .deb and .rpm files). If these are used, then many of the steps described in this README are handled for you automatically. Prerequisites ------------- * Java version 6 or 7 * Tomcat version 6 or 7 * PostgreSQL database version 8 or 9 (preferred) * Perforce Server version 2012.1 or later * The contents of this archive file For the optional Perforce Service Manager (PSM) component: * Ruby 1.8.7 * Rubygems Contents of this archive ------------------------ insights.zip sql create_pa_tables.sql create_template_users.sql upgrade_13_3.sql upgrade_14_1.sql upgrade_14_2.sql upgrade_15_1.sql webapps insights-config.properties insights-log4j.groovy insights.war plugins PLUGINS.html PLUGINS.txt FileExtensionsMetric.groovy FileTypesMetric.groovy HelloWorldMetric.groovy HelloWeatherMetric.groovy HelloSqlMetric.groovy UsageMetric.groovy fileExtensions.properties fileTypes.properties helloWorld.properties helloSql.properties helloWeather.properties usage.properties README.txt README.html 1) Configuring the Database =========================== If you are upgrading from a previous version (either 13.2, 13.3 or 14.X), then skip ahead to section 5, otherwise proceed with sections 1 - 4. Note that upgrades from Insights 13.1 are not supported, and a new install will be required. Insights requires an external database server to manage the statistics data. Currently PostgreSQL is supported (versions 8 and 9 have been tested). You should ensure that both the PostgreSQL server and client are installed. It is possible (for production installations, it is recommended) to install the Postgres server on a different host to Insights, in which case only the client needs to be installed on the Insights host. If you want to run everything on a single host, then both the client and server needs to be installed on the Insights host. On Linux, the client and server are split into two different packages, though installing the server package will generally install the client package as well. To install the server via your package manager: On RedHat, as root, run: yum install postgresql-server (for the server and client) yum install postgresql (for just the client) On Ubuntu, as root, run: apt-get install postgresql (for the server and client) apt-get install postgresql-client (for just the client) To test that the server is correctly installed, you can run: /etc/init.d/postgresql status To test that the client is correctly installed, you can run: sudo -u postgres psql (if the server is local) psql -U postgres -h -W (if the server is remote) On Microsoft Windows, you should download and install the Postgres binaries. The graphical pgAdmin tool is also available on both Windows and Linux, and may be a preferred client if you are not familiar with the command line or PostgreSQL. In order to connect to a remote server, you may need to make some changes to the PostgreSQL configuration. Instructions on how to do this are in the next section. 1.1) Configure PostgreSQL ------------------------- PostgreSQL must be configured to allow TCP connections from the server machine that hosts the Insights web application. Normally, PostgreSQL defaults to restrictive connection options, and these need to be opened up. If you are not familiar with PostgreSQL, see: http://www.postgresql.org/docs/manuals/ For PostgreSQL trust authentication, see: http://www.postgresql.org/docs/8.4/static/auth-methods.html#AUTH-TRUST The following are some of the options you might need to get things running. Consult a PostgreSQL administrator for advice if you need a highly secure environment. * You might need to enable TCP connections within the database configuration. You can often do this by setting listen_addresses = '*' in the postgresql.conf file. * You might need to enable trust authentication for database users. In the pg_hba.conf file, change 'ident' to 'md5' for IPv4 connections from localhost. * You can set a password for the postgres administrator user to allow remote access by running "ALTER USER postgres WITH PASSWORD 'postgres';" * In PostgreSQL 8.4, you might need to run "CREATE LANGUAGE plpgsql;" before proceeding with the following steps. * The PostgreSQL database must be configured to use UTF8. The location of the PostgreSQL configuration files depends on your operating system and the database installation method. 1.2) Create the database and role --------------------------------- Log in to the database as the postgres administrator, either by using the graphical client or via the command line (psql -h localhost -U postgres -W). Run the following commands: CREATE ROLE insights LOGIN PASSWORD 'insights' NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE; CREATE DATABASE insightsdb WITH OWNER insights; This creates a user insights with password insights that can access the database insightsdb You can change any of these values (indeed we suggest that you use a more secure password), but you need to note them so that they are available when you configure the application, and some extra work may be involved when you import the schema. 1.3) Import the schema files into the database ---------------------------------------------- There are five schema files, two which should be used for a new installation of Perforce Insights, and three which are used when upgrading from either 13.2, 13.3 or 14.X. 1.3.1) A New Installation of Insights 15.1 ------------------------------------------ The two schema files create_pa_tables.sql and create_template_user.sql are in the /sql directory. 1. The create_pa_tables.sql file assumes that the database user role is named insights. If this is not the case, you must modify the SQL to change the permission settings. In create_pa_tables.sql, modify each "ALTER TABLE x OWNER TO insights" to alter ownership to the new user. It is safe to perform a search and replace for "OWNER TO insights" to "OWNER TO myuser". 2. Import the two SQL files either using the graphical PostgreSQL client or psql: psql -h localhost -U insights -W insightsdb < create_pa_tables.sql psql -h localhost -U insights -W insightsdb < create_template_user.sql 2) Installing the WAR ===================== The Insights web application consists of a WAR and a configuration file. Insights supports installation of the WAR in a Tomcat application server. Other application servers have not been tested. Note that Insights does not work with any Java version earlier than 6. Java (either the Oracle JDK or OpenJDK) 6 or 7 with Tomcat 6 or 7 are the recommended options. A Java heap size of at least 1.5GB (2GB is preferred) and a PermGen space of 512MB is highly recommended. See the Tomcat documentation for details on configuring this. On Ubuntu, you can change these settings in the following file: /etc/default/tomcat6 (or tomcat7 if you are running that) Look for the first JAVA_OPTS line, and change it to include the following: JAVA_OPTS="-XX:MaxPermSize=512m -Xmx2048m" On RedHat, the file is: /etc/tomcat6/tomcat6.conf Add the following line before the first declaration of JAVA_OPTS: JAVA_OPTS="-XX:MaxPermSize=512m -Xmx2048m" On Windows, check the catalina.bat file, and add the following line: set CATALINA_OPTS="-server -Xms256m -Xmx2048m -XX:MaxPermSize=512m %CATALINA_OPTS%" To install the WAR file and supporting configuration files on Tomcat: 1. Copy the webapps/insights-config.properties file from the ZIP or TGZ file into the classpath for Tomcat. On UNIX, this is often the tomcat/lib directory, which may be /usr/share/tomcat6/lib. 2. If necessary, update the database connection details in insights-config.properties to work with your postgreSQL Insights database. For an upgrade, this file can remain unchanged. 3. Copy webapps/insights.war from the ZIP file or TGZ file to the webapps directory in Tomcat (often /var/lib/tomcat6/webapps on UNIX). The WAR can take a few minutes to deploy, but once it has, it should be accessible via http://hostname:8080/insights 4. If at any point you need to enable debugging in the application, edit the insights-config.properties file and change the value of insights.log.level to debug. Then restart Tomcat. If you use SSL on your Perforce server, then you will also need a .p4trust file in your Tomcat home directory. Normally, Insights will try to create this file automatically if it is needed, but on UNIX systems Tomcat often does not have write permission to this directory. Find the home directory for the Tomcat user (this is often 'tomcat' on RedHat based systems, or tomcat6/tomcat7 on Debian systems). It will be listed in the computer's /etc/passwd file, similar to this: tomcat7:x:118:126::/usr/share/tomcat7:/bin/false In this example, the home directory is /usr/share/tomcat7. Create a file in this directory called .p4trust which is writable by the Tomcat user, e.g.: touch /usr/share/tomcat7/.p4trust chown tomcat7:tomcat7 /usr/share/tomcat7/.p4trust chmod 600 /usr/share/tomcat7/.p4trust This creates an empty file which is only readable/writable by Tomcat. If Insights needs to store trust details for an SSL enabled Perforce Server, then it will be able to use this file. 3) Securing Insights ==================== We strongly recommend that you secure the application server using SSL. Either configure Tomcat to use SSL directly or front Tomcat with an Apache proxy that uses SSL. For more information about using SSL with Tomcat, see: http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html Without the use of SSL, password transmission will not be secure. 4) Configuring Insights ======================= 1. Log in to http://hostname:8080/insights. The first time you log in, you can use 'insightsadmin' with a password of 'P4insightsadmin'. This is the Insights administrator user, which allows you to configure the system for use. 2. Go to the drop-down menu on the top right. There are options for 'Setup', 'Imports', 'Projects', and 'Users'. Each of these options is described below. 4.1) Initialise Database ------------------------ When installing a new system, the first step is to initialise the Perforce data in the database by importing from a Perforce checkpoint. 1. Create a checkpoint from the Perforce Server. If you are unsure about how to create checkpoints, or need a refresher on how they are used, see: http://answers.perforce.com/articles/KB_Article/Checkpoints-and-Journals 2. Select the 'Import' option from the drop-down menu on the top right. This will give you the option to upload a checkpoint to the server, and then import it. 3. In order for the continuous import to work, the location of the Perforce Server must be configured. See the 'Setup' section which follows. When you upload a checkpoint, by default it will be placed within the temporary directory (e.g. /tmp/tomcat-tomcat6-tmp/insights). If you have a large checkpoint file, then you will need to ensure that there is enough space on the disc to upload the file to, though be aware that the file will be copied to Tomcat's work directory first, so there needs to be sufficient space in both locations. Alternatively, you can copy your checkpoint to the upload location on the server manually (using ftp, a shared drive or any other method), and it will be available for import just as if it had been uploaded via the web interface. This is the suggested approach if you have a large checkpoint. Note that everything in the Tomcat temporary directory is deleted when Tomcat is restarted, so if you have a large file that takes a while to upload, you may want to configure the upload location to be somewhere else. The location can be configured from the Setup page, by changing the insights.upload.path. 4.2) Setup ---------- The setup page displays a list of all of the configuration options for Insights. Not all of them are described here - most can be left unchanged. * To edit an option, click the option name and select Edit. * To delete an option, click the option name and select Delete. * To create a new option, click New Configuration. 4.2.1) Perforce Server ---------------------- You must define the Perforce Server instance that will be used for login authentication and streaming. Until you set the Perforce Server information, non-administrative users will be unable to log into Insights using their Perforce user name and password. * insights.p4port - Hostname and port of the Perforce Server. e.g. 'ssl:myserver:1666' * insights.p4user - Username to connect as. This must be a super user. e.g. 'super' * insights.p4passwd - Password or ticket of p4user. e.g. 'supersecretpassword' The streaming import uses "p4 export" to read changes out of the Perforce Server as they happen. If it needs to catch up, then changes are read out of the journals. It is important that any rotated journals are uncompressed, otherwise this data cannot be read. Consult the Perforce administration documentation for information on how to manage checkpoints and journals, but in brief, the -Z (rather than -z) flag will compress checkpoints but leave rotated journals uncompressed. Unless specified with the server configurable 'journalPrefix', journals are expected to be in the P4ROOT. If journals are being moved elsewhere after being rotated, or if a prefix is being specified on the command line, then the streaming process will not be able to find them. If journals are being stored outside of the P4ROOT (which is recommended), then it is important to use the 'journalPrefix' server configurable to control this. If you need further details on how journals can be configured, then see the Perforce System Administrator's Guide: http://www.perforce.com/perforce/doc.current/manuals/p4sag/chapter.backup.html If you are using the optional Perforce Service Manager (PSM) component, you must also set the URL of the web service: * insights.psm.baseURL.main - URL of the PSM web service, in the form http://hostname:port/api/ See section 4.5 below for details. 4.2.2) Jobs ----------- If you are using Perforce jobs, you must specify how to interpret the Perforce jobspec. Since a jobspec can vary widely in how it is structured, and there are no limitations on how the fields are used, you must configure the names of the job fields and their possible values. * insights.jobs.bugs.field.name - The name of the jobspec field that defines the type of job (for example, bug, defect, task, story). Defaults to 'Type'. * insights.jobs.bugs.field.value - The value of the Type field named above that defines a job as being a bug. Defaults to be 'BUG'. * insights.jobs.bugs.severity.name - The name of the jobspec field that defines the severity (or importance) of a bug. Defaults to 'Severity'. * insights.jobs.bugs.status.name - The name of the jobspec field that defines the status of a job (e.g. open, closed, duplicate). Defaults to 'Status'. * insights.jobs.bugs.status.open.value - Values for the 'Status' field that should be interpreted as open. This can contain multiple comma separated values. Defaults to 'open'. * insights.jobs.bugs.status.fixed.value - Values for the 'Status' field that should be interpreted as closed. This can contain multiple comma separated values. Defaults to 'fixed,closed,punted,obsolete,duplicate'. * insights.jobs.bugs.date.reported - The name of the jobspec field that holds the date a job was first opened. Defaults to 'ReportedDate'. * insights.jobs.bugs.date.modified - The name of the jobspec field that holds the date a job was last modified. Defaults to 'ModifiedDate'. You should also specify how job severities are displayed in the graphs. By default, there are three severities configured, and each has two settings. These are: insights.jobs.bugs.values.A.weight=10 insights.jobs.bugs.values.A.color=red insights.jobs.bugs.values.B.weight=3 insights.jobs.bugs.values.B.color=#ff5105 insights.jobs.bugs.values.C.weight=1 insights.jobs.bugs.values.C.color=orange These settings give severity A bugs a weight of 10, severity B bugs a weight of 3, and a severity C bug a weight of 1. They also configure the colors used for these severities when they are displayed on the 'Open Bugs' bar chart. To change the weights or colors, simply edit these values. If you do not use A, B and C for your severities, create new settings for the severities that you do use and delete the existing ones, if necessary. For example, if your severities are CRITICAL, HIGH, MEDIUM, and LOW, then: insights.jobs.bugs.values.CRITICAL.weight=12 insights.jobs.bugs.values.CRITICAL.color=#770000 insights.jobs.bugs.values.HIGH.weight=6 insights.jobs.bugs.values.HIGH.color=#ff0000 insights.jobs.bugs.values.MEDIUM.weight=3 insights.jobs.bugs.values.MEDIUM.color=#ff5105 insights.jobs.bugs.values.LOW.weight=1 insights.jobs.bugs.values.LOW.color=orange 4.3) Project Configuration -------------------------- Initially, Insights does not know how the files in your depot are structured. In order to configure this, log in as 'insightsadmin' and select 'Projects' from the menu at the top right. You will see a list of all currently configured projects. Initially, the list will be empty. Select 'Add Project' for each project, and fill in the details. 4.3.1) Project Name ------------------- This is a descriptive name for the project. 4.3.2) Private & Perforce Group Name ------------------------------------ Normally, information about projects can be accessed by anyone who can log in to Insights. If a project needs restricted access, check this box. A private project can only be accessed by logged-in users who are members of a specific Perforce group. The name of this group is automatically generated from the project name and cannot be set manually. If your project is called "My Project," then the group name is insights_project_my_project. Users who are not members of this group will not be able to select this project from the various filters. However, they will not be restricted from viewing parts of the depot within the project through other means. 4.3.3) Job View --------------- This is the job query that should be used to find jobs (bugs) for this project. The nature of this query depends on how the jobspec is defined. If there is a field named 'project' that holds the name of the project that a bug is assigned to, then the query might look like this: project=projectname For a complete discussion of how to write job queries, see: http://www.perforce.com/perforce/doc.current/manuals/p4guide/08_jobs.html 4.3.4) Branches --------------- The branches of a project define the depot paths that include files associated with the project. A project can have multiple branches, such as MAIN, DEV, or RELEASE, and each branch consists of one or more depot paths. Add a branch, give it a name, and specify each depot path that includes files for that part of the project. 4.3.5) Users & Groups --------------------- Finally, each project can have users associated with it. User filters for a project are limited to those users whom you define as members of the project. If a user isn't associated with a project, then you will not be able to filter by that user, though data about them may still be displayed. If you want to include a large number of users, it is suggested that you set up a group on the Perforce Server and add that to the project. 4.3.6) Generating Statistics ---------------------------- On the main Projects page, there is a link on the bottom right-hand corner to generate statistics for projects. Once a project has been created, a process runs every night to generate statistics for that project. However, when you first create a project, you must run that process for historical data. The Statistics Generation page enables you to select the projects for which you want to generate historical data and the time period you want to cover. The greater the time period, the longer it will take. The time required to generate statistics from historical data depends very much on the amount of that data (number of files, submits, and jobs) in each project. We recommend that you initially generate data for six months for a single project, and then base your estimates for the rest of data generation on the results. Regenerating data for longer time periods for a project can be done at any time. 4.4) User Management ------------------- The Insights User Management page enables you to change the 'insightsadmin' password and set up other users to be administrators or project managers. To access the Insights User Management page, log in as 'insightsadmin' and select 'Users' from the menu at the top right. You can't use this interface to change the password of users authenticated by Perforce, only the internal users insightsadmin and insights_template_user. To find a user, start typing their name into the Username field. Only users who have previously logged into Insights are listed. To modify a user that has not previously logged in, use the Create option (in the toolbar). Creating users using this option does not create them in Perforce and does not allow you to set their password. Once a user has been created, you can edit the user's role, using the Roles tab on the Create User page. A user that has the ROLE_ADMIN role has permission to configure Insights (they see the Setup option on their menu). Users with ROLE_PROJMGR have permission to modify Projects within Insights. They see the Projects option on their menu. 4.5) Perforce Service Manager ----------------------------- The Perforce Service Manager (PSM) is an optional (currently beta) component of Insights that monitors the current performance of a Perforce Server instance. It reports on the current command queue length and shows the most active users and processes using Perforce. The PSM is a standalone component that you must install on the host that runs your Perforce Server, since it monitors the log files from the Perforce Server instance in real time. PSM downloads, installation instructions, and configuration instructions are available on your Insights server here: http://hostname:port/insights/docs/psm Once you have set up PSM, you should ensure that the configuration option insights.psm.baseURL.main has been configured correctly (see 4.2.1). 5) Upgrade from Previous Versions ================================= Upgrades from 13.2 directly to 15.1 are not supported, and you will need to first upgrade to 13.3, then perform the upgrade to 15.1. See the instructions that come with 13.3 on how to do this. If upgrading from 13.3+, then only the following instructions need to be carried out: * Shutdown the Tomcat application server. * Backup the PostgreSQL database. 5.1) Upgrade the Database ------------------------- There are SQL upgrade scripts in the sql directory that perform the upgrade to the named version, so the upgrade_14_1.sql file will upgrade the database to 14.1 from the previous version (13.3). Each of these scripts needs to be run in turn. If you are upgrading from the previous version (14.2), then you only need to run the latest script (which will be upgrade_15_1.sql) The following instructions assume that you are upgrading from 14.2. If you are upgrading from a previous release, then you first need to follow these instructions with the earlier upgrade scripts in order. Once the database has been backed up, you may need to edit the sql file to suit your environment. The upgrade_15_1.sql file assumes that the database owner role is named insights. If this is not the case, you must modify the SQL to change the permission settings. * In upgrade_15_1.sql, modify each "OWNER TO insights" to alter ownership to match your environment. It is safe to perform a search and replace for "OWNER TO insights" to "OWNER TO myuser". You must then run the following command to upgrade the database schema: psql -h localhost -U insights -W insightsdb < upgrade_15_1.sql This should be run in the directory that contains upgrade_15_1.sql or the path to this file specified in the above command. You will be prompted for the database password when this is run. Alternatively, you can load the upgrade file into pgAdmin (or similar graphical tool) and execute it from there. 5.2) Upgrade the WAR -------------------- The configuration for the Insights application is much the same, however the options to control the level of logging have been moved into the insights-config.properties. * Compare the insights-config.properties provided in the new installation, and copy any new options across that you need. * Remove the old insights.war and any exploded directory out of the Tomcat webapps directory. * Copy the new insights.war into the Tomcat webapps directory. * Start up Tomcat. The new version of Insights should deploy itself and after a couple of minutes be ready for use.