NAV Navbar

Integrations API Overview

Integrations in Fibery are quite unusual. It replicates a part of an external app domain and feed data into Fibery and create several Databases.

Dedicated service (integration application) should be implemented to configure and fetch data from an external application.

All communication between integration application and Fibery services is done via standard hypertext protocols, whether it can be HTTP or HTTPS. All integration applications are expected to adhere to a particular API format as outlined in this documentation. The underlying technologies used to develop these integration applications are up to the individual developer.

Users may register their applications with Fibery by providing a HTTP or HTTPS URI for their service. The service must be accessible from the internet in order for the applications gallery to successfully communicate with the service. It is highly recommended that service providers consider utilizing HTTPS for all endpoints and limit access to these services only to IP addresses known to be from Fibery.

In essence, Fibery's applications gallery service acts as a proxy between other Fibery services and the third party provider with some added logic for account and filter storage and validation.

Installed application will be available for all users in your Fibery account. Users from another accounts won't be able to see or use your integration application.

How to add, edit or delete

Let's assume you created an integration app, made it's url available and are ready to try it in Fibery. For example, our holidays app is deployed to Heroku and accessible at https://holidays-fibery.herokuapp.com

Adding custom app

Navigate to space you would like to integrate and click integrate button, find add custom app option and click. Follow the flow.

Add App

Editing custom app

You can change the link to a custom app or force an update to the configuration of the app after deploying it to production by finding your app in catalog and clicking on settings icon in right upper corner.

NOTE: It is recommended to update custom app every time when changes to config or schema of app are happened. Just click update button in described case.

Edit App

Deleting custom app

You can delete a custom app by finding your app in the catalog and clicking on settings icon in right upper corner.

NOTE: In case of app deletion, the databases will not be removed and relations to the databases will not be affected.

Delete App

Domain

App schema

Each app should follow strict schema with the following structure (all fields are required):

Name Type Description Example
name string Name "MyApp"
website string Website "http://myawesomeapp.com"
version string Version "1.0.0"
description string Description "My awesome app"
authentication Array Authentications [{id: "none", name: "no auth"}]
sources Array Empty array []
responsibleFor Object Responsibilities {dataSynchronization: true}

Authentication

Authentication model represents type of authentication in app and has following structure:

Name Type Description Required Example
id string Identity of authenticaiton true "oauth" or "token"
name string Name of authentication true "Connection Settings"
description string Description of authentication false "Give the name for connection"
fields Array Authentication fields false [{optional: false, id: "accountName", name: "Title", type: "text", description: 'Give the name for connection'}]

If your app doesn't require authentication you should add authentication with id = "none". In the case you can omit fields.

Authentication field

Authentication field represents the field in account. So account that will send to API endpoints will consist of the authentication fields. Authentication field has following structure:

Name Type Description Required Example
id string Id of field. The id will be specified in account object. true "accountName"
name string Name of the field. Will be displayed in auth form. true "Title"
type string Type of the field true "text"
description string Description of the field true "Give the name for connection"
optional boolean Is the field optional false false
value string Default value of the field false null

Read about field types. Note that authentication field with some types requires more information to specify. For example, for highlightText type you also need to specify editorMode in authentication field.

Filter

Filter represents filter that will be applied on data. Filter has following structure:

Name Type Description Required Example
id string Id of filter true "table"
title string Title of filter true "Table"
type string Type of filter true "list"
datalist boolean Is filter has values to select from false true
optional boolean Is filter optional false false
secured boolean Secured filter values are not available for change by non-owner false true

Read about filter types. Some additional properties should be specified in case when filters should be displayed in special modes, for example JSON or SQL. For example, for highlightText type you also need to specify editorMode for the filter field.

Fields

Fields in Fibery integration app denote either user-supplied fields or fields within an information schema. There are various types of fields that each represent a different type of data. Fields are also used internally to denote information both required and optional for accounts and filters.

Field Types

The table below lists the types of fields referred to in the Fibery integration app REST API.
The first column denotes the string name of the type as expected in the API, the second column the type of underlying data, the third column any comments and remarks for each field type and the last one show additional options that may be specified.

type datatype comments and remarks Additional options
text string UTF-8 encoding -
number number can be decimal, integer, or float -
bool boolean validates as true/false -
password string only used in account schemas, rendered as a password input -
link - (used for view) show link to external source Need to specify "value" field. It should contain link, for example "http://test.com".
highlightText string text with syntax highlight (json and sql supported for now) Need to specify "editorMode" field. For now "json" and "sql" are supported.
datebox date single date selector
date date date selection with support of Date range grammar -
oauth - (used for view) display OAuth form -
list variant allows to select value from multiple options Optional "datalist_requires" can be specified. For example "datalist_requires" = ["owner"] means that fetching the list depends on "owner" field.
multidropdown Array allows to select multiple values from dropdown list Optional "datalist_requires" can be specified. For example "datalist_requires" = ["owner"] means that fetching the list depends on "owner" field.

A Special Note on Dates

Dates, both with and without times, that are parsed as a result of user-input (through the Date range grammar) are timezone-naive objects. Dates received from connected applications may be either aware or naive of timezones and should represent those dates as strings in an ISO-8601 format.

REST Endpoints

Below are a list of the HTTP endpoints expected in an integration application.

Required

Optional

GET /

{
    "version": "1.0",  // string representing the version of your app
    "name": "My Super Application",  // title of the app
    "description": "All your base are belong to us!", // long description
    "authentication": [], // list of possible account authentication approaches
    "sources": [], // empty error
    "responsibleFor": { // app responsibility
        "dataSynchronization": true // indicates that app is responsible for data synchronization
    }
}

GET "/" endpoint is the main one which returns information about the app. You can find response structure here. Response example:

Authentication Information

{
    "authentication": [
        {
            "id": "basic", // identifier
            "name": "Basic Authentication", // user-friendly title
            "description": "Just using a username and password", // description
            "fields": [  //list of fields to be filled
                {
                    "id": "username",  //field identifier
                    "title": "Username",  //friendly name
                    "description": "Your username, duh!",  //description
                    "type": "text",  //field type (text, password, number, etc.)
                    "optional": true,  // is this a optional field?
                },
                /* ... */
            ]
        }
    ]
}

The authentication object includes all account schema information. It informs the Fibery front-end how to build forms for the end-user to provide required account information. This property is required, even if your application does not require authentication. At least one authentication object must be provided within array.

In case when application doesn't require authentication, the fields array must be omitted. Read more about fields here.

Important note: if your app provides OAuth capabilities for authentication, the authentication identifiers must be oauth and oauth2 for OAuth v1 and OAuth v2, respectively.
Only one authentication type per OAuth version is currently supported.

POST /validate

Incoming body:

{
    "id": "basic", // identifier for the account type
    "fields": { //list of field values to validate according to schema
        "username": "test_user",
        "password": "test$user!",
        /*...*/
    }
}

Success Response:

{"name": "Awesome Account"}

If the account is invalid, the app should return HTTP status 401 (Not Authorized) with a simple JSON object containing an error message:

Failure Response:

{"message": "Your password is incorrect!"}

This endpoint performs account validation when setting up an account for the app and before any actions that uses the account. The incoming payload includes information about the account type to be validated and all fields required:

If the account is valid, the app should return HTTP status 200 with a JSON object containing a friendly name for the account:

Refresh Access Token

Refresh Access Token Request:

{
    "id": "oauth2",
    "fields": {
        "access_token": "xxxx",
        "refresh_token": "yyyy",
        "expire_on": "2018-01-01"
    }
}

Response sample after token refresh:

{
    "name": "Awesome account",
    "access_token": "new-access-token",
    "expire_on": "2020-01-01"
}

In addition this step can be used as a possibility to refresh access token. The incoming payload includes refresh and access token, also it can include expiration datetime. Response should include new access token to override expired one.

POST /api/v1/synchronizer/config

The endpoint returns information about synchronization possibilities based on input parameters. It instructs Fibery about: * Available types * Available filters * Available functionalities

Request

Request example:

{
    "account": {
        "token": "user-token"
    }
}

All input parameters are optional.

Name Type Description
account object selected account's fields

Response

Response example:

{
    "types:": [
        {"id": "bug", "name": "Bug"},
        {"id": "us", "name": "User Story"},    
    ],
    "filters": [
        {
          "id": "modifiedAfter",
          "title": "Modified After",
          "optional": true,
          "type": "datebox"
        }           
    ]
}

Output parameters:

Name Type Description
types [{id: string, name: string}] supported types list with id and display name
filters Array of filters it is used to help the user to exclude non-required data.
webhooks {enabled: true} Optional fields that indicates that webhook functionality is supported

Filter information

The filter object is used to help the user to exclude non-required data. Just like other field-like objects, the filter object is not required. If nothing is provided, users will not be able to filter out data received from the app.

For more information about filters, please refer to domain and fields types.

POST /api/v1/synchronizer/schema

Request example

{
    "types": ["pullrequest", "repository"],
    "filter": {
        "owner": "fibery",
        "repositories": ["fibery/core", "fibery/ui"]
    },
    "account": {
        "token": "token"
    }
}

Integration app must provide data schema in advance so Fibery will be able to create approriate types and relations and then be able to maintain them.

It should provide a schema for all requested types. Each type must contain name and id field. In additional there is a reserved field __syncAction that should be added to the schema if delta synchronization with possibility of removing items should be supported.

Request

Request contains:

Response example:

{
  "repository": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "name": {
      "type": "text",
      "name": "Name"
    },
    "url": {
      "type": "text",
      "name": "Original URL",
      "subType": "url"
    } 
  },
  "pullrequest": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "name": {
      "type": "text",
      "name": "Name"
    },
    "repositoryId": {
      "type": "text",
      "name": "Repository Id",
      "relation": {
        "cardinality": "many-to-one",
        "name": "Repository",
        "targetName": "Pull Requests",
        "targetType": "repository",
        "targetFieldId": "id"
      }
    },
    "__syncAction": {
      "type": "text",
      "name": "Sync Action"
    }
  }
}

Response

Includes schema for all requested types

Schema is JSON object where key is field and value if field description. Field description contains:

field description type
id Field id string
ignore Is field visible in fields catalog boolean
name Field name string
description Field description string
readonly Disable modify field name and type boolean
type Type of field "id", "text" ,"number" , "date", "array[text]"
relation Relation between types see relations section
subType Optional Fibery sub type "url" , "integer", "email", "boolean","html", "md", "files", "date-range"

Relations

Example:

{
  "repository": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "name": {
      "type": "text",
      "name": "Name"
    },
    "url": {
      "type": "text",
      "name": "Original URL",
      "subType": "url"
    } 
  },
  "pullrequest": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "name": {
      "type": "text",
      "name": "Name"
    },
    "repositoryId": {
      "type": "text",
      "name": "Repository Id",
      "relation": {
        "cardinality": "many-to-one",
        "name": "Repository",
        "targetName": "Pull Requests",
        "targetType": "repository",
        "targetFieldId": "id"
      }
    }
  }
}

relation field provides a possibility to create a relation between entities in Fibery. It contains following fields:

field description type
cardinality Type of relation "many-to-one", "many-to-many"/
name Name of the field on source side string
targetType Id of target type string
targetName Field name of target side string
targetFieldId Find relation by value from field string

Repository will have following fields (example includes only relation fields):

Pull Request will have following fields:

Reserved field - __syncAction

Sync action is reserved field that won't be visible for end user. The field is used for delta synchronization when entity should be deleted.

POST /api/v1/synchronizer/data

Data endpoint performs actual data retrieving for the specified integration settings. Data retrieving is run by each type independently.

Data synchronization supports:

Request example:

{
    "requestedType": "pullrequest",
    "types": ["repository", "pullrequest"],
    "filter": {
        "owner": "fibery",
        "repositories": ["fibery/core", "fibery/ui", "fibery/apps-gallery"]
    },
    "account": {
        "token": "token"
    },
    "pagination": {
        "repositories": ["fibery/ui", "fibery/apps-gallery"]
    },
    "lastSynchronizedAt": "2020-09-30T09:08:47.074Z"
}

Request

Inbound payload includes following information:

Response example:

{
    "items": [
        {
            "id": "PR_1231",
            "name": "Improve performance"
        },
        {
            "id": "PR_1232",
            "name": "Fix bugs"
        }
    ],
    "pagination": {
        "hasNext": true,
        "nextPageConfig": {
            "repositories": ["fibery/apps-gallery"]
        }
    },
    "synchronizationType": "full"
}

Response

Outboud payload includes:

Errors

If something goes wrong then integration app should respond with corresponding error HTTP status code and error message.

Sample of error about sync failure:

{
    "message": "Unable to fetch data."
}

But some errors can be fixed if try to fetch data later. In this case error body should include tryLater flag with value true. Fibery will retry this particular page later on.

Sample of error about limits

{
    "message": "Rate limits reached",
    "tryLater": true
}

OPTIONAL

The /logo endpoint is used to provide a SVG representation of a connected application's logo. This endpoint is entirely optional. Valid responses are a HTTP 200 response with a image/svg+xml content type, a HTTP 204 (No Content) response if there is no logo, or a 302 redirect to another URI containing the logo. If no logo is provided, or an error occurs, the application will be represented with our default app logo.

POST /api/v1/synchronizer/datalist

OPTIONAL

Request body:

{
    "types": ["pullrequest", "branch"],
    "account": {
        "token": "token"
    },
    "field": "repository",
    "dependsOn": {
        "owner": "fibery"
    }
}

This endpoint performs retrieving datalists from filter fields that marked with datalist flag.

Request

The inbound payload includes:

Response sample:

{
    "items": [
        {
            "title": "fibery/ui",
            "value": "124"
        },
        {
            "title": "fibery/core",
            "value": "125"
        }
    ]
}

Response

The response from your API should include items that is a JSON-serialized list of name-value objects: The title in each object is what is displayed to the end-user for each value in a combobox and the value is what is stored with the filter and what will be passed with subsequent requests that utilize the user-created filter.

POST /api/v1/synchronizer/filter/validate

OPTIONAL

Request example:

{
    "types": ["repository", "pullrequest"],
    "filter": {
        "owner": "fibery",
        "repositories": ["fibery/core", "fibery/ui", "fibery/apps-gallery"]
    },
    "account": {
        "token": "token"
    }
}

This endpoint performs filter validation. It can be useful when app doesn't know about what filter value looks like. For example, if your app receives sql query as filter you may want to check is that query is valid.

Request

Request body contains:

Error response sample:

{"message": "Your filter is incorrect!"}

Response

If the filter is valid, the app should return HTTP status 200 or 204.

If the account is invalid, the app should return HTTP status 400 (Bad request) with a simple JSON object containing an error message:

POST /api/v1/synchronizer/webhooks

OPTIONAL

The endpoint is responsible for installing, updating or reinstalling webhook based on provided parameters.

Request example:

{
  "types": ["pullrequest", "repository"],
  "filter": {
    "owner": "fibery",
    "repositories": ["fibery/ui", "fibery/core"]
  },
  "account": {
    "token": "token"
  },
  "webhook": null
}

Usage cases:

Request

Request body contains:

Response example:

{
  "id": "webhook_id",
  "events": ["create_repo", "delete_repo"]
}

Response

Response is a JSON object with one required field id. This response object will be send in next attach webhook event.

DELETE /api/v1/synchronizer/webhooks

OPTIONAL

Request example:

{
  "types": ["pullrequest", "repository"],
  "filter": {
    "owner": "fibery",
    "repositories": ["fibery/ui", "fibery/core"]
  },
  "account": {
    "token": "token"
  },
  "webhook": {
    "id": "webhook-id"
  }
}

This endpoint is responsible for uninstalling webhook.

Request

Request body contains:

Error response example:

{"message": "Unable to delete webhook"}

Response

If webhook is deleted then it should return HTTP status 200 or 204.

If something goes wrong then app should respond with corresponding error HTTP status code and error message.

POST /api/v1/synchronizer/webhooks/verify

OPTIONAL

{
  "params": {
    "x-github-id": "1234",
    "x-github-signature256": "dsadsa"
  },
  "payload": {
    "action": "create",
    "repository": {
      "id": "repo1"
    }
  }
}

This route verifies that webhook event is valid and it can be handled. Usually it uses SHA verification. The route should respond with id of webhook.

Request

Request payload includes: * params - event parameters that comes from external system. * payload - event payload that comes from external system.

Success response:

{
  "id": "webhook_id"
}

Response

If verification has passed successfully then response should include webhook id field

Failure example:

{
   "message": "Invalid signature"
}

Otherwise it should reply with error HTTP status code with error message

POST /api/v1/synchronizer/webhooks/transform

OPTIONAL

Request example:

{
  "params": {
      "x-github-id": "1234",
      "x-github-signature256": "dsadsa"
    },
    "payload": {
      "action": "create",
      "repository": {
        "id": "repo1"
      }
    },
    "types": ["pullrequest", "repository", "branch"],
    "filter": {
      "owner": "fibery",
      "repositories": []
    },
    "account": {
      "token": "token"
    }
}

This route is responsible for converting event payload into data that can be handled by Fibery. Data is applied by delta synchronization rules.

Request

Request payload includes:

Response

Response:

{
  "data": {
    "repositories": [
      {
        "id": "repo1",
        "name": "Repo1",
        "__syncAction": "SET"
      }
    ],
    "branches": [
      {
        "id": "master",
        "name": "master",
        "repositoryId": "repo1",
        "__syncAction": "SET"
      }
    ]
  }
}

Response includes data object that is a map of data arrays by type id.

OAuth

if your app provides OAuth capabilities for authentication, the authentication identifiers must be oauth and oauth2 for OAuth v1 and OAuth v2, respectively. Only one authentication type per OAuth version is currently supported.

OAuth v1

POST /oauth1/v1/authorize

The POST /oauth1/v1/authorize endpoint performs obtaining request token and secret and generating of authorization url for OAuth version 1 accounts.

Request body sample:

{
  "callback_uri": "https://oauth-svc.fibery.io/callback?state=xxxxxxx"
}

Included with the request is a single body parameter, callback_uri, which is the redirect URL that the user should be expected to be redirected to upon successful authentication with the third-party service. callback_uri includes query parameter state that MUST be preserved to be able to complete OAuth flow by Fibery.

Response body sample:

{
  "redirect_uri": "https://trello.com/1/OAuthAuthorizeToken?oauth_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&name=TrelloIntegration",
  "token": "xxxx",
  "secret": "xxxx"
}

Return body should include a redirect_uri that the user should be forwarded to in order to complete setup, token and secret are granted request token and secret by third-party service. Replies are then POST'ed to /oauth1/v1/access_token endpoint.

Note: The OAuth implementation requires the account identifier to be oauth for OAuth version 1.

Note: If service provider has callback url whitelisting than https://oauth-svc.fibery.io?state=xxxxx has to be added to the whitelist.

POST /oauth1/v1/access_token

Request body sample:

{
  "fields": {
    "access_token": "xxxx",
    // token value from authorize step
    "access_secret": "xxxxx",
    // secret value from authorize step
    "callback_uri": "https://oauth-svc.fibery.io?state=xxxxx"
  },
  "oauth_verifier": "xxxxx"
}

The POST /oauth1/v1/access_token endpoint performs the final setup and validation of OAuth version 1 accounts. Information as received from the third party upon redirection to the previously posted callback_uri are sent to this endpoint, with other applicable account information, for final setup. The account is then validated and, if successful, the account is returned; if there is an error, it is to be raised appropriately.

The information that is sent to endpoint includes:

Response body sample:

{
  "access_token": "xxxxxx",
  "refresh_token": "xxxxxx",
  "expires_on": "2020-01-01T09:53:41.000Z"
}

Response can include any data that will be used to authenticate account and fetch information.

Tip: You can include parameters with refresh_token and expires_on and then on validate step proceed with access token refresh if it is expired or about to expire.

OAuth v2

POST /oauth2/v1/authorize

Request sample

{
  "callback_uri": "https://oauth-svc.fibery.io",
  "state": "xxxxxx"
}

The POST /oauth2/v1/authorize endpoint performs the initial setup for OAuth version 2 accounts using Authorization Code grant type by generating redirect_uri based on received parameters.

Request body includes following parameters:

Response example:

{
  "redirect_uri": "https://accounts.google.com/o/oauth2/token?state=xxxx&scope=openid+profile+email&client_secret=xxxx&grant_type=authorization_code&redirect_uri=something&code=xxxxx&client_id=xxxxx"
}

Return body should include a redirect_uri that the user should be forwarded to in order to complete setup.
Replies are then POST'ed to /oauth2/v1/access_token endpoint.

Note: The OAuth implementation requires the account identifier to be oauth2 for OAuth version 2.

Note: If service provider has callback url whitelisting than https://oauth-svc.fibery.io has to be added to the whitelist.

POST /oauth2/v1/access_token

Request body sample:

{
  "fields": {
    "callback_uri": "https://oauth-svc.fibery.io"
  },
  "code": "xxxxx"
}

The POST /oauth2/v1/access_token endpoint performs the final setup and validation of OAuth version 2 accounts. Information as received from the third party upon redirection to the previously posted callback_uri are sent to this endpoint, with other applicable account information, for final setup. The account is then validated and, if successful, the account is returned; if there is an error, it is to be raised appropriately.

The information that is sent to endpoint includes:

Response body sample:

{
  "access_token": "xxxxxx",
  "refresh_token": "xxxxxx",
  "expires_on": "2020-01-01T09:53:41.000Z"
}

Response can include any data that will be used to authenticate account and fetch information.

Tip: You can include parameters with refresh_token and expires_on and then on validate step proceed with access token refresh if it is expired or about to expire.

Date range grammar

In order to make filtering by dates and date ranges easier on the end-user, Fibery integration app uses a custom domain specific language to represent dates and date ranges. The syntax allows a user to specify either a static date or a dynamic date range using plain English statements.

Arbitrary Dates

Dates can be easily and quickly specified without any knowledge of the DSL by simply inserting a date in either DD-MMMM-YYYY, DD MMMM YYYY, DD-MM-YYYY or DD MM YYYY format.

The Date Spec DSL also includes various keyword elements that can be used to quickly insert dynamic values. In your date specification you may specify today and yesterday to get the appropriate date value. These date values will always parse in relation to the current day, allowing you to specify a date or date range of, or relative to, these dates.

Periods

Periods in the date spec DSL is where the majority of the magic happens. The DSL understands various time periods, as outlined in the chart below. These time periods can be referenced either aligned to their appropriate boundaries (calendar months, full weeks, quarters, etc.) or arbitrarily aligned to a particular day (the past month, past week, etc.).

Period Aligned value Arbitrary value
day a day a day
week iso week (Mon - Sun) 7 days
month calendar month (1st day start) 30 days
quarter calendar quarter (Q1, Q2, etc.) 90 days
year calendar year (Jan 1 - Dec 31) 365 days

Grammar usage

Periods can be referenced, as mentioned earlier, either aligned to an appropriate date boundary or arbitrarily aligned. When using an aligned period, the period fits to the calendar representation of that period. For example, an aligned month starts on the 1st day and ends on the last, whereas an arbitrary month would represent 30 days starting or ending on a particular day.

LAST and THIS statements

The last keyword creates a period aligned to the last period represented. Consider the following examples:

DSL statement value
last week the last full calendar week starting from Monday
last month` the last full month, starting from the 1st
last quarter the last full quarter
last year the last full year

User can specify period interval exactly. For example, last 3 weeks, last 8 months and etc. this keyword is an alias for last and can be used instead of it.

PREVIOUS statement

The previous keyword creates a range whose starts with first day of the period before the current period and continues for period length. Consider the following examples:

DSL statement value
previous week starts at 12:00:00 AM on the first day of the week before the current week and continues for seven days
previous month starts at 12:00:00 AM on the first day of the month before the current month and continues for all the days of that month
previous quarter starts at 12:00:00 AM on the first day of the calendar quarter before the current calendar quarter and continues to the end of that quarter
previous year starts at 12:00:00 AM on January 1 of the year before the current year and continues through the end of December 31 of that year

User can specify period interval exactly. For example, previous 3 weeks, previous 8 months and etc.

BETWEEN statements

The between keyword allows you to specify a date range or minimum and maximum values to filter on with more power and precision. The full format for this statement is BETWEEN date AND date where both dates are some representation of a date (this can be an arbitrary date, keyword or period). ago keyword can be used inside between to specify date more precisely.

Consider the following examples:

DSL statement value
between 6 quarters ago and 2 months ago starts at 12:00:00 AM on the first day of the calendar quarter 6 quarters before the current calendar quarter and continues until to the first day of the calendar month 2 months before the current calendar month
between 1 January 2016 and today between Jan 1, 2013 and today

FROM and TO statements

from and to allow user to specify ranges without minimum or maximum values.

Consider the following examples:

DSL statement value
from 1 Jan 2010 starts at 1 Jan 2010 till now
to 1 Jan 2012 ends with 1 Jan 2012

Receiving Parsed Dates

The date range grammar parses an input date with each use of a filter that utilizes a date-type field. These dates are provided to application connectors as a dictionary object representing a date range. User-input dates and date ranges are timezone-naive.

Date always is parsed as date range. Even if user enters a single date in a field, this date will be parsed and further represented as a dictionary object with _min equal to start of the date and _max equal to end of the date. If a user enters a date range with no defined minimum or maximum, such as would be the case when using the FROM or TO keywords, _min and _max keys will not be present and the dictionary will have only _min or _max as its single key-value pair.

_min and _max represented as a string in ISO-8601 format.

How to test and debug

Expose local instance

Expose local instance to world:

brew install ngrok
ngrok http 8080

It is possible to run your app on local machine and make the app's url publicly available by using tools like ngrok. Then you will have an ability to debug the app locally after adding it Fibery apps gallery. Don't forget to remove the app from Fibery integration apps catalog after testing.

Integration tests

const request = require(`supertest`);
const app = require(`./app`);
const assert = require(`assert`);
const _ = require(`lodash`);

describe(`integration app suite`, function () {
  it(`should have the logo`, async () => {
    await request(app).get(`/logo`)
      .expect(200)
      .expect(`Content-Type`, /svg/);
  });

  it(`should have app config`, async () => {
    const {body: appConfig} = await request(app).get(`/`)
      .expect(200).expect(`Content-Type`, /json/);

    assert.equal(appConfig.name, `Public Holidays`);
    assert.match(appConfig.description, /public holidays/);
    assert.equal(appConfig.responsibleFor.dataSynchronization, true);
  });

  it(`should have validate end-point`, async () => {
    const {body: {name}} = await request(app).post(`/validate`)
      .expect(200).expect(`Content-Type`, /json/);
    assert.equal(name, `Public`);
  });

  it(`should have synchronization config`, async () => {
    const {body: {types, filters}} = await request(app)
      .post(`/api/v1/synchronizer/config`)
      .expect(200)
      .expect(`Content-Type`, /json/);
    assert.equal(types.length, 1);
    assert.equal(filters.length, 3);
  });

  it(`should have schema holidays type defined`, async () => {
    const {body: {holiday}} = await request(app)
      .post(`/api/v1/synchronizer/schema`)
      .send()
      .expect(200)
      .expect(`Content-Type`, /json/);
    assert.deepEqual(holiday.id, {name: `Id`, type: `id`});
  });

  it(`should return data for CY`, async () => {
    const {body: {items}} = await request(app)
      .post(`/api/v1/synchronizer/data`)
      .send({
        requestedType: `holiday`,
        filter: {
          countries: [`CY`],
        }
      }).expect(200).expect(`Content-Type`, /json/);
    assert.equal(items.length > 0, true);
    const holiday = items[0];
    assert.equal(holiday.id.length > 0, true);
    assert.equal(holiday.name.length > 0, true);
  });

  it(`should return data for BY and 2020 year only`, async () => {
    const {body: {items}} = await request(app)
      .post(`/api/v1/synchronizer/data`)
      .send({
        requestedType: `holiday`,
        filter: {
          countries: [`BY`],
          from: 2020,
          to: 2020
        }
      }).expect(200).expect(`Content-Type`, /json/);
    assert.equal(items.length > 0, true);
    const holidaysOtherThan2020 = _.filter(items, (i) => 
      new Date(i.date).getFullYear() !== 2020);
    assert.equal(holidaysOtherThan2020.length > 0, false);
  });
});

It is recommended to create integration tests before adding your custom app to Fibery apps gallery. Check some tests I created for holidays app.

Tutorial: Simple app

In the tutorial we will implement the simplest app with token authentication. We will use NodeJS and Express framework, but you can use any other programming language. Find source code repository here.

Requirements

const users = {
    token1: {
        name: `Dad Sholler`,
        data: {
            flower: [
                {
                    id: `47fd45cf-5a07-40aa-9ee4-a4258832154a`,
                    name: `Rose`,
                },
                {
                    id: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
                    name: `Lily`,
                },
                {
                    id: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    name: `Tulip`,
                },
            ],
            regionPrice: [
                {
                    id: `56c50696-1d9f-4e4d-9678-448017d25474`,
                    flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    price: 10,
                    name: `Eastern Europe`,
                },
                {
                    id: `503e5efb-7650-4c3a-85e9-f4ebc23adfd5`,
                    name: `Western Europe`,
                    flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    price: 15,
                },
                {
                    id: `87ccd5ef-ed3f-49db-9bcc-c9d1daf91744`,
                    name: `Eastern Europe`,
                    price: 20,
                    flowerId: `47fd45cf-5a07-40aa-9ee4-a4258832154a`,
                },
            ],
        },
    },
    token2: {
        name: `Ben Dreamer`,
        data: {
            flower: [
                {
                    id: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
                    name: `Lily`,
                },
                {
                    id: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    name: `Tulip`,
                },
            ],
            regionPrice: [
                {
                    id: `c3352e8c-e62c-4e26-9a8b-852c1a3d2435`,
                    flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    price: 10,
                    name: `East Coast`,
                },
                {
                    id: `7a581588-d5fc-46aa-a084-8e2b28f3d6e5`,
                    name: `West Coast`,
                    flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
                    price: 15,
                },
                {
                    id: `6e510195-41e3-499b-9a76-9ad928720882`,
                    name: `Asia`,
                    price: 20,
                    flowerId: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
                },
            ],
        },
    },
};

To create simple app you need to implement following endpoints:

Let's implement them one by one. But first let's define dataset

Getting app information

app.get(`/`, (req, res) => {
    res.json({
        id: 'integration-sample-app',
        name: 'Integration Sample App',
        version: `1.0.0`,
        type: 'crunch',
        description: 'Integration sample app.',
        authentication: [
            {
                description: 'Provide Token',
                name: 'Token Authentication',
                id: 'token',
                fields: [
                    {
                        type: 'text',
                        description: 'Personal Token',
                        id: 'token',
                    },
                ],
            },
        ],
        sources: [],
        responsibleFor: {
            dataSynchronization: true,
        },
    });
});

App information should have the structure. Let's implement it. We can see that the app require token authentication and responsible for data synchronization. You can find more information about the endpoint here.

Validate account

app.post(`/validate`, (req, res) => {
    const user = users[req.body.fields.token];
    if (user) {
        return res.json({
            name: user.name,
        });
    }

    res.status(401).json({message: `Unauthorized`});
});

Since authentication is required we should run an authentication and return corresponding user name. You can find more information about the endpoint here.

Getting sync configuration

app.post(`/api/v1/synchronizer/config`, (req, res) => {
    res.json({
        types: [
            {id: `flower`, name: `Flower`},
            {id: `regionPrice`, name: `Region Price`},
        ],
        filters: [],
    });
});

This is basic a scenario and we don't need any dynamic. So we will return a static configuration with no filters. You can find more information about the endpoint here.

Getting schema

const schema = {
   flower: {
       id: {
           name: `Id`,
           type: `id`,
       },
       name: {
           name: `Name`,
           type: `text`,
       },
   },
   regionPrice: {
       id: {
           name: `Id`,
           type: `id`,
       },
       name: {
           name: `Name`,
           type: `text`,
       },
       price: {
           name: `Price`,
           type: `number`,
       },
       flowerId: {
           name: `Flower Id`,
           type: `text`,
           relation: {
               cardinality: `many-to-one`,
               name: `Flower`,
               targetName: `Region Prices`,
               targetType: `flower`,
               targetFieldId: `id`,
           },
       },
   },
};

app.post(`/api/v1/synchronizer/schema`, (req, res) => {
    res.json(
        req.body.types.reduce((acc, type) => {
            acc[type] = schema[type];
            return acc;
        }, {}),
    );
});

We should provide schema for selected types. You can find more information about the endpoint here.

Fetching data

app.post(`/api/v1/synchronizer/data`, (req, res) => {
    const {requestedType, account} = req.body;

    return res.json({
        items: users[account.token].data[requestedType],
    });
});

The endpoint receives requested type, accounts, filter and set of selected types and should return actual data. You can find more information about the endpoint here. And that's it! Our app is ready to use. See full example here.

Tutorial: Holidays App

In this article we will show how to create simple integration app which does not require any authentication. We intend to create a public holidays app which will sync data about holidays for selected countries. The holidays service Nager.Date will be used to retrieve holidays.

App Information

const appConfig = require(`./config.app.json`);
app.get(`/`, (req, res) => res.json(appConfig));

Response (config.app.json):

{
  "id": "holidays-app",
  "name": "Public Holidays",
  "version": "1.0.1",
  "description": "Integrate data about public holidays into Fibery",
  "authentication": [
    {
      "id": "public",
      "name": "Public Access",
      "description": "There is no any authentication required",
      "fields": [
      ]
    }
  ],
  "sources": [
  ],
  "responsibleFor": {
    "dataSynchronization": true
  }
}

Every integration should have the configuration which describes what the app is doing and the authentication methods. The app configuration should be accessible at GET / endpoint and should be publicly available. I used Heroku to host my app which can be found here.

NOTE: All mentioned properties are required.

Since I don't want my app be authenticated I didn't provide any fields for "Public Access" node in authentication. It means that any user will be able to connect their account to the public holidays app. Find an example with token authentication here .

Validate Account

app.post(`/validate`, (req, res) => res.json({name: `Public`}));

This endpoint is responsible for app account validation. It is required to be implemented. Let's just send back the name of account without any authentication since we are creating an app with public access.

Sync configuration

const syncConfig = require(`./config.sync.json`);
app.post(`/api/v1/synchronizer/config`, (req, res) => res.json(syncConfig));

Response(config.sync.json):

{
  "types": [
    {
      "id": "holiday",
      "name": "Public Holiday"
    }
  ],
  "filters": [
    {
      "id": "countries",
      "title": "Countries",
      "datalist": true,
      "optional": false,
      "type": "multidropdown"
    },
    {
      "id": "from",
      "type": "number",
      "title": "Start Year (by default previous year used)",
      "optional": true
    },
    {
      "id": "to",
      "type": "number",
      "title": "End Year (by default current year used)",
      "optional": true
    }
  ]
}

The way data is synchronised should be described.

The endpoint is POST /api/v1/synchronizer/config

types - responsible for describing types which will be synced. For the holidays app it is just one type with id " holidays" and name "Public Holidays". It means that only one integration Fibery database will be created in the space, with the name "Public Holidays".

filters - contains information on how the type can be filtered. In our case there is a multi dropdown ('countries') which is required and marked as datalist. It means that options for this dropdown should be retrieved from app and special end-point should be implemented for that. We have two numeric filters from and to which are optional and can be used to filter holidays by years.

Find information about filters here.

Countries datalist

app.post(`/api/v1/synchronizer/datalist`, wrap(async (req, res) => {
  const countries = await (got(`https://date.nager.at/api/v3/AvailableCountries`).json());
  const items = countries.map((row) => ({title: row.name, value: row.countryCode}));
  res.json({items});
}));

Endpoint POST /api/v1/synchronizer/datalist should be implemented if synchronizer filters has dropdown marked as "datalist": true. Since we have countries multi dropdown filter which should contain countries it is required to implement the mentioned endpoint as well.

For example part of countries response will look like this:

{
  "items": [
    ...
    {
      "title": "Poland",
      "value": "PL"
    },
    {
      "title": "Belarus",
      "value": "BY"
    },
    {
      "title": "Cyprus",
      "value": "CY"
    },
    {
      "title": "Denmark",
      "value": "DK"
    },
    {
      "title": "Russia",
      "value": "RU"
    }
  ]
}

NOTE: For this app, only the list of countries is returned since our config has only one data list. In the case where there are several data lists then we will need to retrieve "field" from request body which will contain an id of the requested list. The response should be formed as an array of items where every element contains title and value properties.

Schema

const schema = require(`./schema.json`);
app.post(`/api/v1/synchronizer/schema`, (req, res) => res.json(schema));

schema.json

{
  "holiday": {
    "id": {
      "name": "Id",
      "type": "id"
    },
    "name": {
      "name": "Name",
      "type": "text"
    },
    "date": {
      "name": "Date",
      "type": "date"
    },
    "countryCode": {
      "name": "Country Code",
      "type": "text"
    }
  }
}

POST /api/v1/synchronizer/schema endpoint should return the data schema of the app. In our case it should contain only one root element "holiday" named after the id of holiday type in sync configuration above. Find the documentation about schemas here.

NOTE: Every schema node should have id and name elements defined.

Data

const getYearRange = filter => {
  let fromYear = parseInt(filter.from);
  let toYear = parseInt(filter.to);

  if (_.isNaN(fromYear)) {
    fromYear = new Date().getFullYear() - 1;
  }
  if (_.isNaN(toYear)) {
    toYear = new Date().getFullYear();
  }
  const yearRange = [];
  while (fromYear <= toYear) {
    yearRange.push(fromYear);
    fromYear++;
  }
  return yearRange;
};

app.post(`/api/v1/synchronizer/data`, wrap(async (req, res) => {
  const {requestedType, filter} = req.body;
  if (requestedType !== `holiday`) {
    throw new Error(`Only holidays database can be synchronized`);
  }
  if (_.isEmpty(filter.countries)) {
    throw new Error(`Countries filter should be specified`);
  }
  const {countries} = filter;
  const yearRange = getYearRange(filter);
  const items = [];
  for (const country of countries) {
    for (const year of yearRange) {
      const url = `https://date.nager.at/api/v3/PublicHolidays/${year}/${country}`;
      console.log(url);
      (await (got(url).json())).forEach((item) => {
        item.id = uuid(JSON.stringify(item));
        items.push(item);
      });
    }
  }
  return res.json({items});
}));

The data endpoint POST /api/v1/synchronizer/data is responsible for retrieving data. There is no paging needed in case of our app, so the data is returned according to selected countries and years interval. The requestedType and filter can be retrieved from the request body. The response should be returned as array in items element.

Tutorial: Notion App

This section is created in order to provide help on creating complex integration app with dynamic data schema, non-primitive data synchronization (for examples files) and oauth2 authentication. The source code (node.js) can be found in official Fibery repository which contains the implementation of integrating Notion databases into Fibery databases. Demo databases can be found here

App Configuration

Route in app.js

app.get(`/`, (req, res) => res.json(connector()));

Returns the description of the app and possible ways to be authenticated in Notion.

connector.config.js:

const config = require(`./config`);
const ApiKeyAuthentication = {
  description: `Please provide notion authentication`,
  name: `Token`,
  id: `key`,
  fields: [
    {
      type: `password`,
      name: `Integration Token`,
      description: `Provide Notion API Integration Token`,
      id: `key`,
    },
    {
      type: `link`,
      value: `https://www.notion.so/help/create-integrations-with-the-notion-api`,
      description: `We need to have your Notion Integration Token to synchronize the data.`,
      id: `key-link`,
      name: `Read how to create integration, grant access and create token here...`,
    },
  ],
};
const OAuth2 = {
  id: 'oauth2',
  name: 'OAuth v2 Authentication',
  description: 'OAuth v2-based authentication and authorization for access to Notion',
  fields: [
    {
      title: 'callback_uri',
      description: 'OAuth post-auth redirect URI',
      type: 'oauth',
      id: 'callback_uri',
    },
  ],
};

const getAuthenticationStrategies = () => {
  return [OAuth2, ApiKeyAuthentication];
};

module.exports.connector = () => ({
  id: `notion-app`,
  name: `Notion`,
  version: config.version,
  website: `https://notion.com`,
  description: `More than a doc. Or a table. Customize Notion to work the way you do.`,
  authentication: getAuthenticationStrategies(),
  responsibleFor: {
    dataSynchronization: true,
  },
  sources: [],
});

As you see there are two authentication ways are defined:

OAuth2

Hardcoded "oauth2" should be used as id in case you would like to implement OAuth2 support in integration app.

OAuth2

Token Authentication

You may use special field type: "link" in order to provide url for external resource where the user can get more info. Use type:"password" for tokens or other text fields which need to be secured.

Token

Token Authorization

Route (app.js):

app.post(`/validate`, (req, res) => promiseToResponse(res, notion.validate(_.get(req, `body.fields`) || req.body)));

Request Body:

{
  "id": "key",
  "fields": {
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "key",
    "owner": "620a3c46cf7154924cf442cb",
    "key": "MY TOKEN",
    "enabled": true
  }
}

Notion call (the name of account is returned):

module.exports.validate = async (account) => {
  const client = getNotionClient(account);
  const me = await client.users.me();
  return {name: me.name}; //reponse should include the name of user account 
};

The implementation of token authentication is the simplest way to implement. We always used it for testing and development since it is not required UI interaction. The request contains id of auth and user provided values. In our case it is key. Other fields are appended by system and can be ignored.

OAuth 2

app.js

app.post('/oauth2/v1/authorize', (req, res) => {
  try {
    const {callback_uri: callbackUri, state} = req.body;
    const redirectUri = oauth.getAuthorizeUrl(callbackUri, state);
    res.json({redirect_uri: redirectUri});
  } catch (err) {
    res.status(401).json({message: `Unauthorized`});
  }
});

app.post('/oauth2/v1/access_token', async (req, res) => {
  try {
    const tokens = await oauth.getAccessToken(req.body.code, req.body.fields.callback_uri);
    res.json(tokens);
  } catch (err) {
    res.status(401).json({message: 'Unauthorized'});
  }
});

OAuth 2 is a bit more complex and requires several routes to be implemented. The POST /oauth2/v1/authorize endpoint performs the initial setup for OAuth version 2 accounts using Authorization Code grant type by generating redirect_uri based on received parameters. Read more here.

The POST /oauth2/v1/access_token endpoint performs the final setup and validation of OAuth version 2 accounts. Information as received from the third party upon redirection to the previously posted callback_uri are sent to this endpoint, with other applicable account information, for final setup. Read more here.

oauth.js

const got = require(`got`);
const CLIENT_ID = process.env.ENV_CLIENT_ID;
const CLIENT_SECRET = process.env.ENV_CLIENT_SECRET;

module.exports = {
  getAuthorizeUrl: (callbackUri, state) => {
    const queryParams = {
      state,
      redirect_uri: callbackUri,
      response_type: 'code',
      client_id: CLIENT_ID,
      owner: `user`,
    };
    const queryParamsStr = Object.keys(queryParams)
      .map((key) => `${encodeURIComponent(key)}=${encodeURIComponent(queryParams[key])}`)
      .join(`&`);
    return `https://api.notion.com/v1/oauth/authorize?${queryParamsStr}`;
  },
  getAccessToken: async (code, callbackUri) => {
    const tokens = await got.post(`https://api.notion.com/v1/oauth/token`, {
      resolveBodyOnly: true,
      headers: {
        "Authorization": `Basic ${Buffer.from(`${CLIENT_ID}:${CLIENT_SECRET}`).toString('base64')}`,
      },
      json: {
        code,
        redirect_uri: callbackUri,
        grant_type: `authorization_code`,
      },
    }).json();
    return {access_token: tokens.access_token};
  },
};

The implementation of oauth is pretty similar for many services and Notion is not exclusion here. Find the code of oauth.js in the right code panel. access_token will be passed into /validate for validating token in future calls.

Synchronizer configuration

app.js (route)

app.post(`/api/v1/synchronizer/config`, (req, res) => {
  if (_.isEmpty(req.body.account)) {
    throw new Error(`account should be provided`);
  }
  promiseToResponse(res, notion.config(req.body));
});

notion.api.js

const getDatabases = async ({account, pageSize = 1000}) => {
  const client = getNotionClient(account);
  let hasNext = true;
  let start_cursor = null;
  const databases = [];
  while (hasNext) {
    const args = {
      page_size: pageSize, filter: {
        value: `database`, property: `object`,
      }
    };
    if (start_cursor) {
      args.start_cursor = start_cursor;
    }
    const {results, has_more, next_cursor} = await client.search(args);
    results.forEach((db) => databases.push(db));
    hasNext = has_more;
    start_cursor = next_cursor;
  }
  return databases;
};

const getDatabaseItem = (db) => {
  const name = _.get(db, `title[0].plain_text`, `Noname`).replace(/[^\w ]+/g, ``).trim();
  return {id: db.id, name};
};

module.exports.config = async ({account, pageSize}) => {
  const databases = await getDatabases({account, pageSize});
  const dbItems = databases.map((db) => getDatabaseItem(db)).concat({id: `user`, name: `User`});
  return {types: dbItems, filters: []};
};

Response example

{
  "types": [
    {
      "id": "f4642444-220c-439d-85d6-378ddff3d510",
      "name": "Features"
    },
    {
      "id": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
      "name": "Tasks"
    },
    {
      "id": "user",
      "name": "User"
    }
  ],
  "filters": []
}

This endpoint returns types which should be synced to Fibery databases. In Notion case it is the list of databases. Static user type is added. Check how the configuration response looks like for Notion Demo.

Databases

Schema of synchronization

app.js (schema route)

app.post(`/api/v1/synchronizer/schema`, (req, res) => promiseToResponse(res, notion.schema(req.body)));

notion.api.js

module.exports.schema = async ({account, types}) => {
  const databases = await getDatabases({account});
  const mapDatabasesById = _.keyBy(databases, `id`);
  const schema = {};
  types.forEach((id) => {
    if (id === `user`) {
      schema.user = userSchema;
      return;
    }
    const db = mapDatabasesById[id];
    if (_.isEmpty(db)) {
      throw new Error(`Database with id "${id}" is not found`);
    }
    schema[id] = createSchemaFromDatabase(db);
  });
  cleanRelationsDuplication(schema);
  return schema;
};

Request example:

{
  "account": {
    "_id": "620a4396aec5dd672c4fed83",
    "access_token": "USER-TOKEN",
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "oauth2",
    "owner": "620a3c46cf7154924cf442cb",
    "enabled": true,
    "name": "Fibery Developer",
    "masterAccountId": null,
    "lastUpdatedOn": "2022-02-21T09:45:37.802Z"
  },
  "filter": {},
  "types": [
    "f4642444-220c-439d-85d6-378ddff3d510",
    "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
    "user"
  ]
}

Response example:

{
  "f4642444-220c-439d-85d6-378ddff3d510": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "archived": {
      "type": "text",
      "name": "Archived",
      "subType": "boolean"
    },
    "created_time": {
      "type": "date",
      "name": "Created On"
    },
    "last_edited_time": {
      "type": "date",
      "name": "Last Edited On"
    },
    "__notion_link": {
      "type": "text",
      "name": "Notion Link",
      "subType": "url"
    },
    "related to tasks (column)": {
      "name": "Related to Tasks (Column) Ref",
      "type": "text",
      "relation": {
        "cardinality": "many-to-many",
        "targetFieldId": "id",
        "name": "Related to Tasks (Column)",
        "targetName": "Feature",
        "targetType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3"
      }
    },
    "tags": {
      "name": "Tags",
      "type": "array[text]"
    },
    "due date": {
      "name": "Due Date",
      "type": "date"
    },
    "name": {
      "name": "Name",
      "type": "text"
    }
  },
  "3bd058e6-a71c-4e9a-8480-a76810ae38d3": {
    "id": {
      "type": "id",
      "name": "Id"
    },
    "archived": {
      "type": "text",
      "name": "Archived",
      "subType": "boolean"
    },
    "created_time": {
      "type": "date",
      "name": "Created On"
    },
    "last_edited_time": {
      "type": "date",
      "name": "Last Edited On"
    },
    "__notion_link": {
      "type": "text",
      "name": "Notion Link",
      "subType": "url"
    },
    "status": {
      "name": "Status",
      "type": "text"
    },
    "assignees": {
      "name": "Assignees Ref",
      "type": "array[text]",
      "relation": {
        "cardinality": "many-to-many",
        "targetType": "user",
        "targetFieldId": "id",
        "name": "Assignees",
        "targetName": "Tasks (Assignees Ref)"
      }
    },
    "specs": {
      "name": "Specs",
      "type": "array[text]",
      "subType": "file"
    },
    "link to site": {
      "name": "Link to site",
      "type": "text",
      "subType": "url"
    },
    "name": {
      "name": "Name",
      "type": "text"
    }
  },
  "user": {
    "id": {
      "type": "id",
      "name": "Id",
      "path": "id"
    },
    "name": {
      "type": "text",
      "name": "Name",
      "path": "name"
    },
    "type": {
      "type": "text",
      "name": "Type",
      "path": "type"
    },
    "email": {
      "type": "text",
      "name": "Email",
      "subType": "email"
    }
  }
}

The schema which describes fields and relations should be provided for each sync type. Find full implementation here. It is not easy thing to implement since we are talking about dynamic data in Notion databases.

It can be noticed that almost any field from Notion database can be mapped into Fibery field using subType attribute. Relations can be mapped as well. Rich text can be sent as html or md by defining corresponding type="text" and subType="md" or "html".

Note: Relation between databases(types) should be declared only once. Double declarations for relations will lead to duplication of relations in Fibery databases. We implemented the function cleanRelationsDuplication in order to remove redundant relation declarations from schema fields.

Files field mapping:

"specs": {
  "name": "Specs",
  "type": "array[text]",
  "subType": "file"
}

Data route

app.js

app.post(`/api/v1/synchronizer/data`, (req, res) => promiseToResponse(res, notion.data(req.body)));

notion.api.js (paging support)

const getValue = (row, {path, arrayPath, subPath = ``}) => {
  let v = null;
  const paths = _.isArray(path) ? path : [path];
  paths.forEach((p) => {
    if (!_.isUndefined(v) && !_.isNull(v)) {
      return;
    }
    v = _.get(row, p);
  });

  if (!_.isEmpty(subPath) && _.isObject(v)) {
    return getValue(v, {path: subPath});
  }

  if (!_.isEmpty(arrayPath) && _.isArray(v)) {
    return v.map((element) => getValue(element, {path: arrayPath}));
  }

  if (_.isObject(v)) {
    if (v.start) {
      return v.start;
    }
    if (v.end) {
      return v.end;
    }
    if (v.type) {
      return v[v.type];
    }
    return JSON.stringify(v);
  }

  return v;
};

const processItem = ({schema, item}) => {
  const r = {};
  _.keys(schema).forEach((id) => {
    const schemaValue = schema[id];
    r[id] = getValue(item, schemaValue);
  });
  return r;
};

const resolveSchema = async ({pagination, client, requestedType}) => {
  if (pagination && pagination.schema) {
    return pagination.schema;
  }
  if (requestedType === `user`) {
    return userSchema;
  }
  return createSchemaFromDatabase(await client.databases.retrieve({database_id: requestedType}));
};

const createArgs = ({pageSize, pagination, requestedType}) => {
  const args = {
    page_size: pageSize,
  };
  if (!_.isEmpty(pagination) && !_.isEmpty(pagination.start_cursor)) {
    args.start_cursor = pagination.start_cursor;
  }
  if (requestedType !== `user`) {
    args.database_id = requestedType;
  }
  return args;
};

module.exports.data = async ({account, requestedType, pageSize = 1000, pagination}) => {
  const client = getNotionClient(account);
  const schema = await resolveSchema({pagination, client, requestedType});
  const args = createArgs({pageSize, pagination, requestedType});
  const data = requestedType !== `user`
    ? await client.databases.query(args)
    : await client.users.list(args);
  const {results, next_cursor, has_more} = data;
  return {
    items: results.map((item) => processItem({account, schema, item})),
    "pagination": {
      "hasNext": has_more,
      "nextPageConfig": {
        start_cursor: next_cursor,
        schema: has_more ? schema : null,
      },
    },
  };
};

Request example:

{
  "filter": {},
  "types": [
    "f4642444-220c-439d-85d6-378ddff3d510",
    "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
    "user"
  ],
  "requestedType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
  "account": {
    "_id": "620a4396aec5dd672c4fed83",
    "access_token": "USER-TOKEN",
    "app": "620a3c9baec5dd25794fed7a",
    "auth": "oauth2",
    "owner": "620a3c46cf7154924cf442cb",
    "enabled": true,
    "name": "Fibery Developer",
    "masterAccountId": null,
    "lastUpdatedOn": "2022-02-21T13:30:51.350Z"
  },
  "lastSynchronizedAt": null,
  "pagination": null
}

Response example:

{
  "items": [
    {
      "id": "4455580b-000b-4313-8128-f1ca2d2dec34",
      "archived": false,
      "created_time": "2022-02-14T11:28:00.000Z",
      "last_edited_time": "2022-02-14T11:30:00.000Z",
      "__notion_link": "https://www.notion.so/Login-Page-4455580b000b43138128f1ca2d2dec34",
      "related to tasks (column)": [
        "b829daf3-bae5-40a0-a090-56a30f240a28"
      ],
      "tags": [
        "Urgent"
      ],
      "due date": "2022-02-24",
      "name": [
        "Login Page"
      ]
    },
    {
      "id": "9b3dff11-582b-498a-ba9b-571827ab3ca7",
      "archived": false,
      "created_time": "2022-02-14T11:28:00.000Z",
      "last_edited_time": "2022-02-14T11:29:00.000Z",
      "__notion_link": "https://www.notion.so/Home-Page-9b3dff11582b498aba9b571827ab3ca7",
      "related to tasks (column)": [
        "987a714b-0b7e-4b03-bdaf-c0efc5d522fb",
        "539a4d0e-6871-434b-a5cb-619f5bd5a911"
      ],
      "tags": [
        "Important",
        "Urgent"
      ],
      "due date": "2022-02-14",
      "name": [
        "Home Page"
      ]
    }
  ],
  "pagination": {
    "hasNext": false,
    "nextPageConfig": {
      "start_cursor": null,
      "schema": null
    }
  }
}

Notion supports paged output, so it is handy to fetch data page by page. The response should include pagination node with hasNext equals to true or false and nextPageConfig (next page configuration) which will be included with the future request as pagination.

You may notice that we have included schema into nextPageConfig (pagination config). It is not required and it is done as an optimization in order to save some between pages fetching on schema resolving. In other words the pagination can be used as a context cache between page calls.

Source Code

The source code of Notion integration can be found in our public repository as well as other examples. Notion app is used in production and can be tried by following integrate link in your database editor.