Live journal system

This section describes how the Helix Search journaling system tracks when tasks are added to a queue and when they are completed.

Overview

Each queue in the system has its journal file. The naming convention for a journal file is the queue's name with a .jnl extension, such as IndexQueue.jnl. By default, journal files are located in the .jnl/ directory, for example, /opt/perforce/helix-p4search/jnl. You can customize your journal preferences using the com.perforce.p4search.index.journal-path configuration.

When a task is added to the queue, a journal entry is created, and a second entry is made when the task is completed. This process is similar to double-entry bookkeeping. If the queue needs to be restored, the journal is replayed. Entries to add and remove tasks cancel each other out, leaving only uncompleted tasks that can be added back to the queue.

Startup

When the system starts, it checks for the existence of a journal file. If the file exists, it is rotated and moved to a new file with a .jnl.tmp extension. The rotated journal file is then processed, with the add (+) job entries being read into an ordered map. Entries with a minus (-) sign remove the corresponding job from the map. Any remaining jobs in the map are added to the active queue and logged in the journal. After this process, the rotated journal is deleted, and the system startup continues.

Rotation

The journal is rotated and compacted every hour if the number of completed tasks (entries) exceeds the threshold defined by com.perforce.p4search.index.journal-threshold. Compacted journal entries are saved to a file with the extension .ckp. While the results are computed in memory and written back to the compacted journal .ckp file, a temporary .ckp.tmp file is used. If a compacted journal file is present at system startup, it is processed along with the rotated journal.

Journals

Name Description

IndexQueue.jnl

When a user submits new files, they are added to the IndexQueue.jnl.

BackgroundQueue.jnl

Background tasks, such as long indexing, are sent to the BackgroundQueue.jnl.

RenderQueue.jnl

Render tasks processed by an external service are placed in the RenderQueue.jnl.

Journal schema

Action Type Separator Task ID Separator TaskData

+

<Task symbol>

:

<Task ID>

:

BASE64 encoded {JSON}

-

<Task symbol>

:

<Task ID>

<empty>

<empty>

Schema types

Type Symbol Task ID Example

ChangeIndexTask

c

Change number

0000000

FileIndexTask

f

Depot revision specifier

//depot/main/foo.txt#1

AssetTask

a

Depot revision specifier

//depot/main/foo.txt#1
RevisionIndexTask r

Depot revision specifier

//depot/main/foo.txt#1

RenderTask

m

Depot revision specifier

//depot/main/foo.txt#1

UserIndexTask u

User name

ballison

Example code

+c:917343:eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJjaGFuZ2UiOjkxNzM0MywiZm9yY2UiOnRydWUsInRhc2tJRCI6ImM6OTE3MzQzIn0=
+f:917283-//depot/main/p4-perl/P4/Progress.pm#4:eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJmaWxlIjp7Im1hcHBlZCI6ZmFsc2UsImhlYWRDaGFuZ2UiOjAsImhlYWRSZXYiOjAsImhhdmVSZXYiOjAsImRpZ2VzdCI6IkJERjc1QkUxMkQwMEUxQ0M3RjZBNEUzNUZDMDYyMkM4IiwiZmlsZVNpemUiOjM4OTUsIm9wZW5DaGFuZ2VsaXN0SWQiOjAsInVucmVzb2x2ZWQiOmZhbHNlLCJyZXNvbHZlZCI6ZmFsc2UsInJlcmVzb2x2YWJsZSI6ZmFsc2UsIm90aGVyTG9ja2VkIjpmYWxzZSwib3RoZXJBY3Rpb25MaXN0IjpbXSwib3RoZXJDaGFuZ2VsaXN0IjpbXSwib3RoZXJPcGVuTGlzdCI6W10sInNoZWx2ZWQiOmZhbHNlLCJyZXNvbHZlUmVjb3JkcyI6W10sImF0dHJpYnV0ZXMiOnt9LCJwcm9wYWdhdGluZ0F0dHJpYnV0ZXMiOnt9LCJhdHRyaWJ1dGVUeXBlcyI6e30sIm9wU3RhdHVzIjoiVkFMSUQiLCJnZW5lcmljQ29kZSI6MCwic2V2ZXJpdHlDb2RlIjowLCJyYXdDb2RlIjowLCJ1bmlxdWVDb2RlIjowLCJzdWJDb2RlIjowLCJzdWJTeXN0ZW0iOjAsImRlcG90UGF0aCI6eyJwYXRoU3RyaW5nIjoiLy9kZXBvdC9tYWluL3A0LXBlcmwvUDQvUHJvZ3Jlc3MucG0iLCJwYXRoVHlwZSI6IkRFUE9UIn0sImZpbGVUeXBlIjoidGV4dCIsInN0YXJ0UmV2aXNpb24iOi0xLCJlbmRSZXZpc2lvbiI6NCwiY2hhbmdlTGlzdElkIjo5MTcyODMsImFjdGlvbiI6IkVESVQiLCJ1c2VyTmFtZSI6InBhbGxlbiIsImJhc2VSZXYiOi0xLCJ1bm1hcCI6ZmFsc2UsImVuZEZyb21SZXYiOi0xLCJzdGFydEZyb21SZXYiOi0xLCJlbmRUb1JldiI6LTEsInN0YXJ0VG9SZXYiOi0xLCJ3b3JrUmV2IjotMSwibG9ja2VkIjpmYWxzZSwic2hlbHZlZENoYW5nZSI6LTEsInN5bmNUaW1lIjowLCJyYXdGaWVsZHMiOnt9LCJyZWZyZXNoYWJsZSI6ZmFsc2UsInVwZGF0ZWFibGUiOmZhbHNlLCJEQVRFX0ZPUk1BVCI6Inl5eXkvTU0vZGQgSEg6bW06c3MifSwiY2hhbmdlbGlzdCI6eyJjaGFuZ2UiOjkxNzI4MywidGltZSI6MTQ2MTY3NzE1NzAwMCwidXNlciI6InBhbGxlbiJ9LCJmb3JjZSI6ZmFsc2UsInRhc2tJRCI6ImY6OTE3MjgzLS8vZGVwb3QvbWFpbi9wNC1wZXJsL1A0L1Byb2dyZXNzLnBtIzQifQ==
-f:917283-//depot/main/p4-perl/P4/Progress.pm#4

Understanding the code example

Here is an explanation of the code example outlining the activities they perform:

  • The following code adds entry for ChangeIndexTask which is written when a task is added:

    eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJjaGFuZ2UiOjkxNzM0MywiZm9yY2UiOnRydWUsInRhc2tJRCI6ImM6OTE3MzQzIn0=
    {"index":{"index":"alias-perforce1"},"change":917343,"force":true,"taskID":"c:917343"}
  • The following code adds entry for FileIndexTask which is written when a task is added.

    eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJmaWxlIjp7Im1hcHBlZCI6ZmFsc2UsImhlYWRDaGFuZ2UiOjAsImhlYWRSZXYiOjAsImhhdmVSZXYiOjAsImRpZ2VzdCI6IkJERjc1QkUxMkQwMEUxQ0M3RjZBNEUzNUZDMDYyMkM4IiwiZmlsZVNpemUiOjM4OTUsIm9wZW5DaGFuZ2VsaXN0SWQiOjAsInVucmVzb2x2ZWQiOmZhbHNlLCJyZXNvbHZlZCI6ZmFsc2UsInJlcmVzb2x2YWJsZSI6ZmFsc2UsIm90aGVyTG9ja2VkIjpmYWxzZSwib3RoZXJBY3Rpb25MaXN0IjpbXSwib3RoZXJDaGFuZ2VsaXN0IjpbXSwib3RoZXJPcGVuTGlzdCI6W10sInNoZWx2ZWQiOmZhbHNlLCJyZXNvbHZlUmVjb3JkcyI6W10sImF0dHJpYnV0ZXMiOnt9LCJwcm9wYWdhdGluZ0F0dHJpYnV0ZXMiOnt9LCJhdHRyaWJ1dGVUeXBlcyI6e30sIm9wU3RhdHVzIjoiVkFMSUQiLCJnZW5lcmljQ29kZSI6MCwic2V2ZXJpdHlDb2RlIjowLCJyYXdDb2RlIjowLCJ1bmlxdWVDb2RlIjowLCJzdWJDb2RlIjowLCJzdWJTeXN0ZW0iOjAsImRlcG90UGF0aCI6eyJwYXRoU3RyaW5nIjoiLy9kZXBvdC9tYWluL3A0LXBlcmwvUDQvUHJvZ3Jlc3MucG0iLCJwYXRoVHlwZSI6IkRFUE9UIn0sImZpbGVUeXBlIjoidGV4dCIsInN0YXJ0UmV2aXNpb24iOi0xLCJlbmRSZXZpc2lvbiI6NCwiY2hhbmdlTGlzdElkIjo5MTcyODMsImFjdGlvbiI6IkVESVQiLCJ1c2VyTmFtZSI6InBhbGxlbiIsImJhc2VSZXYiOi0xLCJ1bm1hcCI6ZmFsc2UsImVuZEZyb21SZXYiOi0xLCJzdGFydEZyb21SZXYiOi0xLCJlbmRUb1JldiI6LTEsInN0YXJ0VG9SZXYiOi0xLCJ3b3JrUmV2IjotMSwibG9ja2VkIjpmYWxzZSwic2hlbHZlZENoYW5nZSI6LTEsInN5bmNUaW1lIjowLCJyYXdGaWVsZHMiOnt9LCJyZWZyZXNoYWJsZSI6ZmFsc2UsInVwZGF0ZWFibGUiOmZhbHNlLCJEQVRFX0ZPUk1BVCI6Inl5eXkvTU0vZGQgSEg6bW06c3MifSwiY2hhbmdlbGlzdCI6eyJjaGFuZ2UiOjkxNzI4MywidGltZSI6MTQ2MTY3NzE1NzAwMCwidXNlciI6InBhbGxlbiJ9LCJmb3JjZSI6ZmFsc2UsInRhc2tJRCI6ImY6OTE3MjgzLS8vZGVwb3QvbWFpbi9wNC1wZXJsL1A0L1Byb2dyZXNzLnBtIzQifQ==			
    {"index":{"index":"alias-perforce1"},"file":{"mapped":false,"headChange":0,"headRev":0,"haveRev":0,"digest":"BDF75BE12D00E1CC7F6A4E35FC0622C8","fileSize":3895,"openChangelistId":0,"unresolved":false,"resolved":false,"reresolvable":false,"otherLocked":false,"otherActionList":[],"otherChangelist":[],"otherOpenList":[],"shelved":false,"resolveRecords":[],"attributes":{},"propagatingAttributes":{},"attributeTypes":{},"opStatus":"VALID","genericCode":0,"severityCode":0,"rawCode":0,"uniqueCode":0,"subCode":0,"subSystem":0,"depotPath":{"pathString":"//depot/main/p4-perl/P4/Progress.pm","pathType":"DEPOT"},"fileType":"text","startRevision":-1,"endRevision":4,"changeListId":917283,"action":"EDIT","userName":"pallen","baseRev":-1,"unmap":false,"endFromRev":-1,"startFromRev":-1,"endToRev":-1,"startToRev":-1,"workRev":-1,"locked":false,"shelvedChange":-1,"syncTime":0,"rawFields":{},"refreshable":false,"updateable":false,"DATE_FORMAT":"yyyy/MM/dd HH:mm:ss"},"changelist":{"change":917283,"time":1461677157000,"user":"pallen"},"force":false,"taskID":"f:917283-//depot/main/p4-perl/P4/Progress.pm#4"}
  • Delete entry for FileIndexTask. This is written when a task is completed.

Clearing journal queues

If you want to restart your Helix Core Server without processing journal files on start-up, you can delete the following journal files:

  • IndexQueue.jnl

  • BackgroundQueue.jnl

  • RenderQueue.jnl

By default, the journal files are located in the .jnl/ directory, for example, /opt/perforce/helix-p4search/jnl.

If these journal files are present on start-up, they will be processed. However, if they are deleted before restarting, the Helix Core Server will not process any old journal files.