projectdoc Toolbox

Tips on specifying search queries for Lucene. This also applies to projectdoc's query macros.

The projectdoc Toolbox uses Lucene for searching documents. This document provides some tips on how to formulate queries and extensions to searches provided by projectdoc.

Use Exact Match Queries

 

Searches with Lucene are by design lenient. In case your are looking for a term, Lucene is capable of applying grammar rules to not run exact match searches. While this is great for the typical full text search, in case you are matching with properties you may want to be strict. For these use cases the projectdoc Toolbox provides the Exact Match Query.

Exact Match Queries are the recommended way of specifying queries.

Select Clauses in Display Macros

Select a property by specifying the name of the property.

Select the value of the Name property

 

Name

Deep Links in Select Clause

Since version 2.0 the projectdoc Toolbox also supports Deep Links as an experimental feature. If the property name references a property whose value is a link, a deep link (specified with "->") allows to traverse the document tree.

Deep Links may have deeper depth than 1.

Deep Links with Depth > 1

 

Ref->Person->Address->Phone

Note that prior to version 2.5 of the projectdoc Toolbox only single valued properties can be used with deep links. So if you are using an older version, if more than one link is found in a property value, only the first will be traversed.

Macros supporting Deep Links:

Display Document Properties Macro
Renders a template with property references.
Display List Macro
Lists references to projectdoc documents in a list. List contain names and optional short descriptions.
Display List Template Macro
Lists references to projectdoc documents in a list. List items are defined by templates referencing properties.
Display Table Macro
Lists references to projectdoc documents in a table. Allows to select document properties for columns. Also non-list representations are provided.
Index Card Macro
Renders transcluded content fetched from documents of a result set.
Index Entries Table Macro
Renders a table of index entries.
Tour Macro
Renders a predefined list of documents in a table.
Tour-by-Property Macro
Renders a predefined list of documents in a table . Documents are selected by a document property. Allows to select document properties for columns. Also non-list representations are provided.
Transclude Documents Macro
Renders transcluded content fetched from documents of a result set.
Transclusion Macro
Transcludes content from a document marked with the content marker macro.
Transclusion Reference Macro
Transcludes content via a reference from a document marked with the content marker macro.

Where Clauses in Display Macros

The where clause of display macros allows to add a constraint on the documents to display in the result set.

Property Names

To use property names in Lucene searches, the property names must not contain blanks. So if the search is restricted to user stories with just one story point, use this:

StoryPoints = "1" 

Template authors may use the $[...] marker to translate the property name to its normalized form:

$[Story Points] = "1" 

This is especially useful if the labels are provided by localized resource bundles. In this case the template author has not to bother, if the name contains spaces or not.

Exact Match Queries recommended

 

We would recommend to run Exact Match Queries per default on property values.

The example above does not use it so that we do not explain two concepts here.

Reference Property Values

If the result set should contain only documents where one property matches the value of a property of the current document, use the ${...} placeholder to reference the property in the current document.

$[Story Points] = ${Magic Value}

For the example, the result set will only contain documents, that have the Story Point property set to a value matching that of the Magic Value property of the document the search is part of.

Exact Match Queries recommended

 

We would recommend to run Exact Match Queries per default on property values.

The example above does not use it so that we do not explain two concepts here.

Curly Braces

 

Curly braces may cause problems on some instances of Confluence when used in a string parameter field of a macro. This is especially true (and reproducible) if you use an opening and an immediate closing bracket like this: "{}".

This is a known issue (CONFSERVER-33399) and is also discussed in Do curly braces in string macro parameters break the macro?

To work around this problem you may escape the curly braces with a backslash as in

$[Story Points] = $\{Magic Value\}

You need to use this workaround if you cannot save the page (as described in the issue above). Otherwise it is just a failed rendering of the macro in the macro editor.

This issue is solved since version 5.0.

Exact Match Queries

If the result should be an exact match query, use the following pattern:

$<Story Points> = [Magic Value]

In case the value is not a constant, you may also use a reference to the property of the document:

$<Story Points> = [${Magic Value}]
 

The syntax $<...> is read as: Please run the preprocessor of projectdoc over the value before it is passed to Lucene.

Escaping Special Characters

Since version 1.9 it is possible to escape special characters with a backslash (\).

$<Name> = [My Name \(with brackets\)]

To escape the escape character also use the escape character.

$<Name> = [My Name \\ containing a slash]

Deep Links in Where Clause

Since version 2.0 of the projectdoc Toolbox Deep Links are supported for property references (on the right side) as an experimental features.

 
$<Story Points> = [${Master->Ref Story Points}]

Note that you cannot use deep links on the left side of the where clause without Materialization. Since version 4.5 this is supported by property control mat, by space property Materialize by Doctype, or by Doctype Descriptor. Materialization is also possible in prior versions of the projectdoc Toolbox with a little more verbose approach. Read Materialize Properties for more information on this.

Subtree Queries

If you need to constrain the search result to documents with a given ancestor, you can take advantage of artificial properties like AncestorNames and AncestorIds.

$<AncestorNames> = [This Root Page]

Or you could constrain the search result by the current page.

$<AncestorIds> = [${Page ID}]
 

See Subtree Queries for examples.

Ancestor Queries

For some type of properties it is useful to query for a given value or its ancestors. E.g. if you want to have all documents relevant for the Quick Response Team (QRT), a role derived from Ops, you may want to collect all documents marked with either QRT or Ops.

$<Audience> = ^QRT^

This is not only a shortcut for not listening all parents. If you add the following to the role template, the document will show all documents relevant for the role:

$<Audience> = ^${Name}^

Note that in the above example Audience is a property of a document that refers to documents of type Role. The hierarchy is defines on role documents. It may be confusing for new uses if they assume that the hierarchy is defined on the document containing the the Audience property.

If you need to search the hierarchy down the tree, use "°" instead of "^". So "°" stands for downstream, "^" for upstream search.

 

Ancestor queries are based on the property information in the blueprint of a document type. Document authors cannot define hierarchies based on document instances. It is the job of template authors to do this.

If you encounter unexpected results, you may want to refresh your caches, especially the doctype cache. Please refer to Cache Refresh Actions for details.

 

See Ancestor Queries for examples.

List Queries

2.0

 

List queries are supported since version 2.0.

List queries run an exact match on a property where one of the values in the list has to match.

$<Name> ~ (One, Two, Three)

expands to

$<Name>=[One] OR $<Name>=[Two] OR $<Name>=[Three]

This is especially useful when a value is provided as a property.

$<Name> ~ (${Some Names})

Note that the List of values found in Some Names may have links or be a macro (such as the Name List Macro). Values with a comma are not allowed since the comma is the delimiter of the list elements.

Useful Examples

Here are some more examples on using queries in various use cases.

Prefix Query

To select all resource documents that are read by a specific user ("jd" in this example) in a given year (specified by the document property Readjd in the format yyyy-MM-dd), use the prefix query:

ReadBy = "jd" AND Readjd = 2014*

Querying List Values

The Tags Property is one that is supported by all document types. It specifies a comma-separated list of values.

To select on those documents by tag values, use this:

Tags = (("test-me" AND "continous-delivery") OR "My-Tag")

Search all Topic Types that are no Fragments

A fragment is a basic document a couple of topic types (table, example, ...) derive from. To select only documents of a type that is not a fragment or one of its subtypes, use the following:

+TypeAncestors:[* TO *] AND -TypeAncestors:Fragment 

Note that in this case all documents have to have their topic type specific. If those that have not should also be part of the result set, use:

+Doctype:topic AND -TypeAncestors:Fragment

Using Dates

Since version 1.10 the projectdoc Toolbox provides normalized date value to be used with Lucene searches. If the property value is specified with the Confluence Date Autocompletion (opened with the shortcut '//'), and for the creation and last modification date, a normalized date is provided as an artificial property.

The name of this artificial property is that of the original property plus the §-sign appended.

Here is an example where the creation date is check against a custom property named Last Update.

$[Creation Date§]: [${Last Update§} TO NOW]

There is also an artificial property value containing the date as a timestamp in milliseconds, prefixed with zeros up to 19 digits (%019d). The name of the artificial property is constructed from the original name plus the suffix Timestamp (separated by a blank). Note that the creation timestamp is called Creation Timestamp (not Creation Date Timestamp).

 

Appending the $-sign instead of the §-sign will render the text representation of the date.

Using Date Ranges

Confluence provides a couple of search fields to help finding the desired documents.

created:[20150311 TO 20150317]
created:[20150311 TO NOW]

Ranges work with Date Functions. Use the normalized date value to define range boundaries.

Date§: [20190101 TO ${now("", "-6w")}]
Date§: [${now("", "-6w")} TO ${now("", "-4w")}]

The Date property is recommended to be specified with the Confluence Date Autocompletion (opened with the shortcut '//').

Search Functions

A query function is a virtual function call that will replace the call with the return value of the function.

 

Search functions are provided since version 3.2 of the projectdoc Toolbox.

The syntax for using search functions is:

$<PROPERTY NAME>=${function-name(argument-list)}

Functions can be used for a value in a Where clause parameter of a projectdoc macro.

Functions cannot be used for a property name or outside the Where clause.

 

Here are some usage examples:

$<Start Date§>=${now()}

$<Login Name>=${currentUser()}

$<Attendee>=${currentUser("fn")}

Supported Functions

The following functions are supported:

  • currentUser()
  • now()
  • startOfDay()
  • endOfDay()
  • startOfWeek()
  • endOfWeek()
  • startOfMonth()
  • endOfMonth()
  • startOfYear()
  • endOfYear()

User Functions

There is only one user function that provides access to the current user of a session.

The currentUser function supports the format argument.

Valid formats are

  • id (default) - login name (id) of the current user
  • fn - Full Name of current user

If no user is currently logged in to the current session, then the format id returns "#anonymous" and the format fn returns "#Anonymous".

 

The following calls will return the login name of the currently logged in user.

currentUser()
currentUser("")
currentUser("id")

In case the full name, as provided in the user's profile is requested, use:

currentUser("fn")

Date Functions

The date functions provide a date in the format "yyyyMMdd". This way it can easily be compared to date properties by the use of the normalized date representation (add '§' to the name of the property like "Start Date§").

The following functions are provided:

NameDescription

now

The current date at the time the page is rendered.

startOfDayThe date at the first millisecond on the current day.
endOfDayThe date at the last millisecond on the current day.
startOfWeek

The date at the first millisecond of the first day of the week the current day is part of.

Note that currently you cannot control the definition of the first day.

endOfWeek

The date at the last millisecond of the last day of the week the current day is part of.

Note that currently you cannot control the definition of the last day.

startOfMonthThe date at the first millisecond of the first day of the month the current day is part of.
endOfMonthThe date at the last millisecond of the first day of the month the current day is part of.
startOfYearThe date at the first millisecond of the first day of the year the current day is part of.
endOfYearThe date at the last millisecond of the first day of the year the current day is part of.

Implementation Details

 

Note that now is calculated per macro. So in case you have multiple macros on a page, the "now" point in time is when the macro is called. For most use cases this should not make any differences. In case a call is sent some milliseconds before midnight, two macros may calculate different days.

The date functions allow two arguments.

  • format
  • increment 

If you want to specify the increment, but not the format, use "" (empty string) as the first argument.

Abbreviated Date Function Call Form

 

Since version 4.1 the abbreviated form without format argument is also supported.

Simply use only the second argument without double quotes.

$<Date§> >= ${now(-2w)}
 

The following calls are semantically identically.

$<Date§> >= ${startOfMonth()}
$<Date§> >= ${startOfMonth("")}
$<Date§> >= ${startOfMonth("", "")}

The following calls are semantically identically, the first only available in 4.1 and above.

$<Date§> <= ${startOfMonth(-1d)}  // 4.1 and above
$<Date§> <= ${startOfMonth("", "-1d")}

Unsupported operators: '<' and '>'

 

Prior to version 4.1 the operators '>' and '<' are not supported!

Please use '>=' and '<=' instead.

Format

Specify the format as defined for the Joda Time date formatter: DateTimeFormat.

Typically use:

FormatDescription

y

Year. Typically use "yyyy" for four digits (i.e. 2019).

MMonth. Typically use "MM" for two digits (i.e. 03 and 12).
ddDay. Typically use "dd" for two digits (i.e. 01 and 15).
H

Hours.  Typically use "HH" for two digits (i.e. 08 and 23).

m

Minutes.  Typically use "mm" for two digits (i.e. 01 and 59).

The default format is: "yyyyMMdd".

Render Format

 

Note that the format is used to render the date. If the date is compared to another date the compare is not executed on the date object, but on the rendered date. Therefore the format must be comparable lexicographically. Any format would do that shows the year first, then the month (leading zero in case the month ordinal number is smaller ten, and lastly the day (again with leading zero).

Compate Date with Format

 

Assume a document property named Date with a date value in text form: 17.11.2019

To match that date on November the 23rd in 2019 the function requires to subtract 6 days.

$<Date> = ${now("dd.MM.yyyy", "-6d")}

In order to match the date value, the format must be specified. This is because the rendered dates are compared, not the parsed date objects.

Increment

With the second argument to the date function you may alter the date.

The format is:

(+/-)nn(y|M|w|d|h|m)

Use + (default) or - to add or subtract from the calculated value.

Provide the amount of units at 'nn'. If omitted, one is assumed.

Add the unit identifier:

IDDescription

y

Years

MMonths
wWeeks
dDays
H

Hours

m

Minutes

For instance:

 
+2d

5y

-10w

-2d10y
 

Here are some usage examples:

$<Date§> >= ${now("", "-2w")}
$<Date§> >= ${endOfMonth("", "1w")}
$<Date§> >= ${startOfMonth("", "+3d")}

Using date ranges:

Date§: [${now("", "-6w")} TO ${now("", "-4w")}]

$<Date§> >= ${now("", "-6w")} AND $<Date§> <= ${now("", "-4w")}

Abbreviated Date Function Call Form

 

Since version 4.1 the abbreviated form without format argument is also supported.

Instead of

$<Date§> >= ${now("", "-2w")}
$<Date§> >= ${endOfMonth("", "1w")}
$<Date§> >= ${startOfMonth("", "+3d")}

simply write

$<Date§> >= ${now(-2w)}
$<Date§> >= ${endOfMonth(1w)}
$<Date§> >= ${startOfMonth(+3d)}

Note that there are no double quotes around the increment specification.

Using Checkboxes

You can select property values that are checkbox and use checkboxes in your where clauses.

The checked checkbox has a text value of complete, an unchecked checkbox has a value of incomplete.

See Tasks as Property Values for an example.

Using Not

In general Not in a query is only allowed when it follows an positive expression.

To specify a document property not to have a certain value, use a range query like this:

+ID:[* TO *] AND -ID:6555694

Searching for an empty attribute using Not

Searching all Documents having an attribute that is not set e.g. not having a parent. This is especially useful, if you want to list all root documents of a given document type (see Parent Property of projectdoc Documents for an example using the Parent Property).

*:* -Parent:[* TO *]

or

*:* NOT Parent:[* TO *]

The *:* is only needed if you have no positive criteria. A Lucene query must not start with "NOT".

 

If you need to have all documents in the result that actually have a parent, then use

Parent:[* TO *]

Note that you can only select on properties, not on sections.

 

If you want to make sure that a document with a specific ID is not part of the result set, use this:

Tags = "logging" NOT ID = 3704077

This does not do the trick (the syntax is wrong):

Tags = "logging" AND ID NOT 3704077

Document Properties for Selection

The Display All Document Properties Macro provides access to the properties and artificial properties a document provides in tabular form.

Mind Stopwords!

Suppose you have two documents with names

  • Querying in apples
  • Querying apples

Here are the query results:

Query StringResultExplanation
Name = "Querying in apples"Only: Querying applesThe stopword 'in' is removed from the query string.
Name = "Querying apples"Only: Querying applesThe document containing the 'in' does not match the given search string.
Name = (Querying in apples)Both: Querying apples and Querying in applesFinds all containing 'Querying' AND 'apples', 'in' is removed since it is a stopword.
Name = (Querying apples)Both: Querying apples and Querying in applesFinds all containing 'Querying' AND 'apples'.
$<Name> = [Querying in apples]Only: Querying in applesAn exact match that checks for the exact phrase.
$<Name> = [Querying apples]Only: Querying applesAn exact match that checks for the exact phrase.

Resources

Search
Information on how to create queries to conduct searches.
Confluence Search Syntax
Describes the special words and punctuation marks you can use to refine your search.
Confluence Search Fields
Gives an overview of the Apache Lucene search fields used in Confluence.
Lucene Query Parser Syntax (4.3.0)
Information on the query parser syntax from the API documentation.
Tips
List of tips to use Confluence with the projectdoc Toolbox and fun! Tips address users of different experience levels.