Live journal system

This section describes how the Helix Search journaling system tracks when tasks are added to a queue and when they are completed.

Overview

Each queue in the system has its journal file. The naming convention for a journal file is the queue's name with a .jnl extension, such as IndexQueue.jnl. By default, journal files are located in the .jnl/ directory, for example, /opt/perforce/helix-p4search/jnl. You can customize your journal preferences using the com.perforce.p4search.index.journal-path configuration.

When a task is added to the queue, a journal entry is created, and a second entry is made when the task is completed. This process is similar to double-entry bookkeeping. If the queue needs to be restored, the journal is replayed. Entries to add and remove tasks cancel each other out, leaving only uncompleted tasks that can be added back to the queue.

Startup

When the system starts, it checks for the existence of a journal file. If the file exists, it is rotated and moved to a new file with a .jnl.tmp extension. The rotated journal file is then processed, with the add (+) job entries being read into an ordered map. Entries with a minus (-) sign remove the corresponding job from the map. Any remaining jobs in the map are added to the active queue and logged in the journal. After this process, the rotated journal is deleted, and the system startup continues.

Rotation

The journal is rotated and compacted every hour if the number of completed tasks (entries) exceeds the threshold defined by com.perforce.p4search.index.journal-threshold. Compacted journal entries are saved to a file with the extension .ckp. While the results are computed in memory and written back to the compacted journal .ckp file, a temporary .ckp.tmp file is used. If a compacted journal file is present at system startup, it is processed along with the rotated journal.

Journal schema

Action Type Separator Task ID Separator TaskData

+

<Task symbol>

:

<Task ID>

:

BASE64 encoded {JSON}

-

<Task symbol>

:

<Task ID>

<empty>

<empty>

Schema types

Type Symbol Task ID Example

ChangeIndexTask

c

Change number

0000000

FileIndexTask

f

Depot revision specifier

//depot/main/foo.txt#1

AssetTask

a

Depot revision specifier

//depot/main/foo.txt#1
RevisionIndexTask r

Depot revision specifier

//depot/main/foo.txt#1

RenderTask

m

Depot revision specifier

//depot/main/foo.txt#1

UserIndexTask u

User name

ballison

Example code

+c:917343:eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJjaGFuZ2UiOjkxNzM0MywiZm9yY2UiOnRydWUsInRhc2tJRCI6ImM6OTE3MzQzIn0=
+f:917283-//depot/main/p4-perl/P4/Progress.pm#4:eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJmaWxlIjp7Im1hcHBlZCI6ZmFsc2UsImhlYWRDaGFuZ2UiOjAsImhlYWRSZXYiOjAsImhhdmVSZXYiOjAsImRpZ2VzdCI6IkJERjc1QkUxMkQwMEUxQ0M3RjZBNEUzNUZDMDYyMkM4IiwiZmlsZVNpemUiOjM4OTUsIm9wZW5DaGFuZ2VsaXN0SWQiOjAsInVucmVzb2x2ZWQiOmZhbHNlLCJyZXNvbHZlZCI6ZmFsc2UsInJlcmVzb2x2YWJsZSI6ZmFsc2UsIm90aGVyTG9ja2VkIjpmYWxzZSwib3RoZXJBY3Rpb25MaXN0IjpbXSwib3RoZXJDaGFuZ2VsaXN0IjpbXSwib3RoZXJPcGVuTGlzdCI6W10sInNoZWx2ZWQiOmZhbHNlLCJyZXNvbHZlUmVjb3JkcyI6W10sImF0dHJpYnV0ZXMiOnt9LCJwcm9wYWdhdGluZ0F0dHJpYnV0ZXMiOnt9LCJhdHRyaWJ1dGVUeXBlcyI6e30sIm9wU3RhdHVzIjoiVkFMSUQiLCJnZW5lcmljQ29kZSI6MCwic2V2ZXJpdHlDb2RlIjowLCJyYXdDb2RlIjowLCJ1bmlxdWVDb2RlIjowLCJzdWJDb2RlIjowLCJzdWJTeXN0ZW0iOjAsImRlcG90UGF0aCI6eyJwYXRoU3RyaW5nIjoiLy9kZXBvdC9tYWluL3A0LXBlcmwvUDQvUHJvZ3Jlc3MucG0iLCJwYXRoVHlwZSI6IkRFUE9UIn0sImZpbGVUeXBlIjoidGV4dCIsInN0YXJ0UmV2aXNpb24iOi0xLCJlbmRSZXZpc2lvbiI6NCwiY2hhbmdlTGlzdElkIjo5MTcyODMsImFjdGlvbiI6IkVESVQiLCJ1c2VyTmFtZSI6InBhbGxlbiIsImJhc2VSZXYiOi0xLCJ1bm1hcCI6ZmFsc2UsImVuZEZyb21SZXYiOi0xLCJzdGFydEZyb21SZXYiOi0xLCJlbmRUb1JldiI6LTEsInN0YXJ0VG9SZXYiOi0xLCJ3b3JrUmV2IjotMSwibG9ja2VkIjpmYWxzZSwic2hlbHZlZENoYW5nZSI6LTEsInN5bmNUaW1lIjowLCJyYXdGaWVsZHMiOnt9LCJyZWZyZXNoYWJsZSI6ZmFsc2UsInVwZGF0ZWFibGUiOmZhbHNlLCJEQVRFX0ZPUk1BVCI6Inl5eXkvTU0vZGQgSEg6bW06c3MifSwiY2hhbmdlbGlzdCI6eyJjaGFuZ2UiOjkxNzI4MywidGltZSI6MTQ2MTY3NzE1NzAwMCwidXNlciI6InBhbGxlbiJ9LCJmb3JjZSI6ZmFsc2UsInRhc2tJRCI6ImY6OTE3MjgzLS8vZGVwb3QvbWFpbi9wNC1wZXJsL1A0L1Byb2dyZXNzLnBtIzQifQ==
-f:917283-//depot/main/p4-perl/P4/Progress.pm#4

Understanding the code example

Here is an explanation of the code example outlining the activities they perform:

  • The following code adds entry for ChangeIndexTask which is written when a task is added:

    eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJjaGFuZ2UiOjkxNzM0MywiZm9yY2UiOnRydWUsInRhc2tJRCI6ImM6OTE3MzQzIn0=
    {"index":{"index":"perforce1"},"change":917343,"force":true,"taskID":"c:917343"}
  • The following code adds entry for FileIndexTask which is written when a task is added.

    eyJpbmRleCI6eyJpbmRleCI6InBlcmZvcmNlMSJ9LCJmaWxlIjp7Im1hcHBlZCI6ZmFsc2UsImhlYWRDaGFuZ2UiOjAsImhlYWRSZXYiOjAsImhhdmVSZXYiOjAsImRpZ2VzdCI6IkJERjc1QkUxMkQwMEUxQ0M3RjZBNEUzNUZDMDYyMkM4IiwiZmlsZVNpemUiOjM4OTUsIm9wZW5DaGFuZ2VsaXN0SWQiOjAsInVucmVzb2x2ZWQiOmZhbHNlLCJyZXNvbHZlZCI6ZmFsc2UsInJlcmVzb2x2YWJsZSI6ZmFsc2UsIm90aGVyTG9ja2VkIjpmYWxzZSwib3RoZXJBY3Rpb25MaXN0IjpbXSwib3RoZXJDaGFuZ2VsaXN0IjpbXSwib3RoZXJPcGVuTGlzdCI6W10sInNoZWx2ZWQiOmZhbHNlLCJyZXNvbHZlUmVjb3JkcyI6W10sImF0dHJpYnV0ZXMiOnt9LCJwcm9wYWdhdGluZ0F0dHJpYnV0ZXMiOnt9LCJhdHRyaWJ1dGVUeXBlcyI6e30sIm9wU3RhdHVzIjoiVkFMSUQiLCJnZW5lcmljQ29kZSI6MCwic2V2ZXJpdHlDb2RlIjowLCJyYXdDb2RlIjowLCJ1bmlxdWVDb2RlIjowLCJzdWJDb2RlIjowLCJzdWJTeXN0ZW0iOjAsImRlcG90UGF0aCI6eyJwYXRoU3RyaW5nIjoiLy9kZXBvdC9tYWluL3A0LXBlcmwvUDQvUHJvZ3Jlc3MucG0iLCJwYXRoVHlwZSI6IkRFUE9UIn0sImZpbGVUeXBlIjoidGV4dCIsInN0YXJ0UmV2aXNpb24iOi0xLCJlbmRSZXZpc2lvbiI6NCwiY2hhbmdlTGlzdElkIjo5MTcyODMsImFjdGlvbiI6IkVESVQiLCJ1c2VyTmFtZSI6InBhbGxlbiIsImJhc2VSZXYiOi0xLCJ1bm1hcCI6ZmFsc2UsImVuZEZyb21SZXYiOi0xLCJzdGFydEZyb21SZXYiOi0xLCJlbmRUb1JldiI6LTEsInN0YXJ0VG9SZXYiOi0xLCJ3b3JrUmV2IjotMSwibG9ja2VkIjpmYWxzZSwic2hlbHZlZENoYW5nZSI6LTEsInN5bmNUaW1lIjowLCJyYXdGaWVsZHMiOnt9LCJyZWZyZXNoYWJsZSI6ZmFsc2UsInVwZGF0ZWFibGUiOmZhbHNlLCJEQVRFX0ZPUk1BVCI6Inl5eXkvTU0vZGQgSEg6bW06c3MifSwiY2hhbmdlbGlzdCI6eyJjaGFuZ2UiOjkxNzI4MywidGltZSI6MTQ2MTY3NzE1NzAwMCwidXNlciI6InBhbGxlbiJ9LCJmb3JjZSI6ZmFsc2UsInRhc2tJRCI6ImY6OTE3MjgzLS8vZGVwb3QvbWFpbi9wNC1wZXJsL1A0L1Byb2dyZXNzLnBtIzQifQ==			
    {"index":{"index":"perforce1"},"file":{"mapped":false,"headChange":0,"headRev":0,"haveRev":0,"digest":"BDF75BE12D00E1CC7F6A4E35FC0622C8","fileSize":3895,"openChangelistId":0,"unresolved":false,"resolved":false,"reresolvable":false,"otherLocked":false,"otherActionList":[],"otherChangelist":[],"otherOpenList":[],"shelved":false,"resolveRecords":[],"attributes":{},"propagatingAttributes":{},"attributeTypes":{},"opStatus":"VALID","genericCode":0,"severityCode":0,"rawCode":0,"uniqueCode":0,"subCode":0,"subSystem":0,"depotPath":{"pathString":"//depot/main/p4-perl/P4/Progress.pm","pathType":"DEPOT"},"fileType":"text","startRevision":-1,"endRevision":4,"changeListId":917283,"action":"EDIT","userName":"pallen","baseRev":-1,"unmap":false,"endFromRev":-1,"startFromRev":-1,"endToRev":-1,"startToRev":-1,"workRev":-1,"locked":false,"shelvedChange":-1,"syncTime":0,"rawFields":{},"refreshable":false,"updateable":false,"DATE_FORMAT":"yyyy/MM/dd HH:mm:ss"},"changelist":{"change":917283,"time":1461677157000,"user":"pallen"},"force":false,"taskID":"f:917283-//depot/main/p4-perl/P4/Progress.pm#4"}
  • Delete entry for FileIndexTask. This is written when a task is completed.