Async events API
Fetch API
Storage API

Rate this page:

Forge’s EAP offers experimental features to selected users for testing and feedback purposes. These features are not supported or recommended for use in production environments. They are also subject to change without notice.

For more information about EAPs, see What's coming.

Complex queries (EAP)

On a simple key/value store, the query builder API lets you use the startsWith condition to filter results.

When your app uses custom entities, you'll have multiple attributes assigned to your keys. This provides you with more options for storing and structuring your app data. In turn, the query builder API provides additional methods and parameters for building complex queries against your custom entities.

Before you begin

Complex queries and custom entities capabilities are available with latest version of the Forge CLI. To upgrade the Forge CLI, completely remove the Forge CLI and install it again:

1
2
npm uninstall -g @forge/cli
npm i -g @forge/cli@latest

After upgrading your Forge CLI version, our team will need to enable both complex queries and custom entities on your app. To do that, we need your app's ID.

Submit your app's ID through this form. We will notify you once we've enabled complex queries and custom entities on your app. When that happens, you should be able to deploy and install your app on development and staging environments.

Custom entities

Custom entities are user-defined data structures for storing app data. Forge's storage API lets you query data stored in these structures using a wide array of query conditions. These query conditions make it possible to build advanced, complex queries to suit your app's operations.

Custom entities are keys with multiple typed or untyped attributes. You can define attributes with the following data types:

  • string
  • integer
  • float
  • boolean
  • any

Custom entities are defined in your manifest.yml as part of the storage property. Each custom entity also includes an indexes section where you define your query's filter patterns (more on this later). The storage property is a child of app, and uses the following syntax:

1
2
storage:
  entities:
    - name: <custom entity name>
      attributes:
        <attribute1>: 
          type: <type>
        <attribute2>: 
          type: <type>
        <attributeN>: 
          type: <type>
      index: 
        - <attributeN>
        - <attributeN>

Entities cannot be deleted. We are working on implementing the capabilities to do this in a future update.

Limits

Custom entities are subject to several limits, namely:

  • For each app, you can only set 5 custom entities (with a maximum of 50 attributes each). A custom entity name should:

    • be a minimum of 3 characters
    • be a maximum of 60 characters
    • follow the regex /^[a-zA-Z0-9]+$/
    • not be empty
  • Each entity can have up to 50 attributes. An attribute name should:

    • be a minimum of 1 character
    • be a maximum of 30 characters
    • follow the regex /^[A-Za-z][_0-9A-Za-z]*$/
    • not be empty

Indexes

While the entities property assigns multiple attributes to each key, indexes sets which attributes to create indexes for. Attributes with indexes are optimized for your queries; as such, you should create indexes based on the query patterns you intend to use.

With this EAP release, indexes cannot be deleted. Existing indexes cannot be updated either.

We are working on implementing the capabilities to do both in a future update.

Index types

You can declare either a simple or named index.

A simple index specifies one attribute (which you can use to reference the index in your queries):

1
2
indexes:
  - <attribute>

A named index allows you to optimize for more complex query patterns. Named indexes use the following optional parameters:

  • partition: optimizes your index for exact matches
  • range: optimizes your index for the use of query conditions
  • name: used to reference the index in your queries

Your index name:

  • should be a minimum of 3 characters
  • maximum of 50 characters
  • should follow the regex /^[a-zA-Z0-9]+([\w:-][.]{0,1})*[a-zA-Z0-9]+$/
  • should not be empty

The range and partition parameters can be used together or alone; both accept all data types (except for any). If you use either or both, you must set a name. You can set multiple attributes for partition, but only one attribute for range.

1
2
indexes:
  - name: <value>
    range: 
      - <attribute>
    partition: 
      - <attribute1>
      - <attribute2>
      - <attribute3>

You can set a maximum of 5 indexes per app.

Deploying apps with indexes

If your app uses indexes, the forge deploy command will send an indexing request to Forge's hosted storage service. This storage service will then create or update indexes as necessary while your app's code is deployed. The indexing process's duration scales with your data set's size, from a minimum of 5 minutes for small data sets.

The indexing process is independent of the rest of the deployment. As such, the forge deploy command will normally complete while the indexing process is still ongoing. Until the indexing process completes, you won't be able to install your app on any sites.

To check the status of the indexing process on an environment (namely, development or staging), run:

1
2
forge storage entities indexes list -e <environment>

The indexing process is complete once all indexes have an ACTIVE status.

Basic methods

All complex queries operate on a custom entity's index. They follow the same basic signature:

1
2
await storage
  .entity("<custom-entity>")
  .query()
  .index()

This structure contains all the required methods for a complex query. The entity method sets which custom entity to query, and index sets which of those entity's indexes to query. Each query can only target one index from one custom entity.

When using indexes that feature a partition, you must specify a value to match the parameter's attribute:

1
2
await storage
  .entity("<custom-entity>")
  .query()
  .index("<index-name>", {
    partition: ["<value>"]
  })

If your index's partition has multiple attributes, then you must set a value for each attribute. In addition, you must also set each value in the order they are declared in the index. For example, consider the following index:

1
2
indexes:
  - name: by-gender-and-age
    partition: 
      - gender
      - age

An appropriate query for this would be:

1
2
await storage
  .entity("employee")
  .query()
  .index("by-gender-and-age", {
    partition: ["male",20]
  })

Filtering methods

You can use the following optional methods to filter and sort data matches:

where

While index lets you filter matches to an index's partition, the where method lets you filter against an index's range:

1
2
.where(WhereConditions.<condition>("<value>"))

Here, WhereConditions allows you to use any of several conditions to filter your results even further.

andFilter / orFilter

You can only use the index and where methods once per query. To add more conditions to your query, use either of the following methods:

  • andFilter: all conditions must be matched.
    1
    2
    .andFilter("<attribute>",FilterConditions.<condition>("<value>"))
    
  • orFilter: only one condition must be matched.
    1
    2
    .orFilter("<attribute>",FilterConditions.<condition>("<value>"))
    

Both methods use FilterConditions to set a wide variety of conditions.

Within the same query, you can use multiple andFilter and orFilter methods. However, you cannot use both methods within the same query.

In addition, the andFilter and orFilter methods are in-memory filters. Using them can sometimes produce pages with no results, with cursor pointing to the next page where actual results exist.

sort

The sort method displays your results in either ascending (ASC) or descending (DESC) order:

1
2
.sort(SortOrder.<"ASC|DESC">)

By default, results are displayed in ascending order.

Conditions

The following conditions are available on where, andFilter, and orFilter methods:

  • beginsWith
  • between
  • equalsTo
  • isGreaterThan, isLessThan
  • isGreaterThanOrEqualTo, isLessThanOrEqualTo

The following conditions can only be used for andFilter and orFilter :

  • exists, doesNotExist
  • contains, doesNotContain
  • notEqualsTo

Example entity

The following manifest.yml excerpt shows a custom entity named employee with several attributes and indexes:

1
2
app:
  id: "ari:cloud:ecosystem::app/406d303d-0393-4ec4-ad7c-1435be94583a"

  storage:
    entities:
      - name: employee
        attributes:
          surname: 
            type: string
          age: 
            type: integer
          employmentyear: 
            type: integer
          gender: 
            type: string
          nationality: 
            type: string
        indexes:
          - surname
          - employmentyear
          - name: by-age
            range: 
              - age
          - name: by-age-per-gender
            partition: 
              - gender
            range: 
              - age  

This entity also creates four indexes based on the following employee attributes:

  • surname
  • employmentyear
  • age (further optimized for filtering according to different age ranges)
  • age per gender (further optimized for filtering according to age ranges for each gender)

Example queries

Using the previous section's example entity and its indexes, the following queries demonstrate the use of each method:

Example queryDescription
await storage
  .entity("employee")
  .query()
  .index("surname")
  .getMany()
Targets the surname index of the employee entity.
await storage
  .entity("employee")
  .query()
  .index("by-age")
  .where(WhereConditions.isGreaterThan(30))
  .sortOrder(Sort."DESC")
  .getMany()

Targets the by-age index, which uses age as its range. From this, the where method will limit matches to employees above the age of 30.

Results will be displayed in descending order.

await storage
  .entity("employee")
  .query()
  .index("by-age-per-gender", {
    partition: ["female"]
  })
  .getMany()
Targets the by-age-per-gender index, and will limit matches to female employees.
await storage
  .entity("employee")
  .query()
  .index("by-age-per-gender", {
    partition: ["female"]
  })
  .where(WhereConditions.isGreaterThan(30))
  .getMany()
Targets the by-age-per-gender index, which also uses age as its range. From this, the where method limit matches only to female employees above the age of 30.
await storage
  .entity("employee")
  .query()
  .index("by-age-per-gender", {
    partition: ["female"]
  })
  .where(WhereConditions.isGreaterThan(30))
  .andFilter("employmentyear",
    FilterConditions.isGreaterThan(2020))
  .andFilter("nationality",
    FilterConditions.equalsTo("Australian"))
  .getMany()
Using the by-age-per-gender index, limits matches only to female Australian employees above the age of 30 who were also hired after 2020.

Example app

We also released an app template for testing out complex queries and custom entities. You can clone and test this app's code from this Bitbucket repository. Refer to this repository's README for more information.

Rate this page: