Syncing Data

Overview

When syncing data with Merge, we recommend a combination of webhooks and polling.

Save your users' account token

Get the account token from the linking process for an embedded Merge Link. Learn more in our embedded Merge Link guide.

You can also use the Linked Account linked webhook to get the account token. See the example payload below and learn how to configure Merge Webhooks in our guide.

To authenticate your API requests to Merge, save your users’ account token in your database. You will need the account token to poll for data in step 4.

Example Linked Account linked webhook payload

1 {
2   "hook": {
3     "id": "e8affe31-8ae0-4b37-8c50-d86303094dc4",
4     "event": "LinkedAccount.linked",
5     "target": "https://webhook.site/105606a8-cfa3-4415-bae0-954e25137f00"
6   },
7   "linked_account": {
8     "id": "4ac10f37-c656-4e9a-89a1-1b04f9e9a343",
9     "integration": "Ashby",
10     "integration_slug": "ashby",
11     "category": "ats",
12     "end_user_origin_id": "12345678910",
13     "end_user_organization_name": "Example Organization",
14     "end_user_email_address": "jack.cavalier@merge.dev",
15     "status": "COMPLETE",
16     "webhook_listener_url": "https://api.merge.dev/api/integrations/webhook-listener/abcd1234defg5678",
17     "is_duplicate": false,
18     "account_type": "PRODUCTION"
19   },
20   "data": {
21     "account_token": "{ACCOUNT-TOKEN}",
22     "is_relink": false
23   }
24 }

Sync data when Merge emits a sync notification webhook

We recommend using the Linked Account synced webhooks to manage sync activities at scale. Whenever you receive a sync notification webhook for a Linked Account, start pulling data and kick off the logic in step 3.

Important fields:

Field	Description
`hook.event`	The event type that triggered the webhook. See our webhooks guide for more information.
`linked_account.id`	The ID of the associated Linked Account.
`data.sync_status`	Handle edge cases when `last_sync_result` is `FAILED` or `PARTIALLY SYNCED`. See our Help Center article on sync statuses.

Example Linked Account synced webhook payload

1 {
2   "hook": {
3     "id": "e8affe31-8ae0-4b37-8c50-d86303094dc4",
4     "event": "LinkedAccount.sync_completed",
5     "target": "https://webhook.site/105606a8-cfa3-4415-bae0-954e25137f00"
6   },
7   "linked_account": {
8     "id": "4ac10f37-c656-4e9a-89a1-1b04f9e9a343",
9     "integration": "Ashby",
10     "integration_slug": "ashby",
11     "category": "ats",
12     "end_user_origin_id": "12345678910",
13     "end_user_organization_name": "Example Organization",
14     "end_user_email_address": "jack.cavalier@merge.dev",
15     "status": "COMPLETE",
16     "webhook_listener_url": "https://api.merge.dev/api/integrations/webhook-listener/abcd1234defg5678",
17     "is_duplicate": false,
18     "account_type": "PRODUCTION"
19   },
20   "data": {
21     "is_initial_sync": true,
22     "integration_name": "Ashby",
23     "integration_id": "ashby",
24     "sync_status": {
25       "ats.Candidate": {
26         "last_sync_finished": "2023-12-29T18:57:12Z",
27         "last_sync_result": "PARTIALLY_SYNCED"
28       },
29       "ats.Application": {
30         "last_sync_finished": "2023-12-29T18:59:25Z",
31         "last_sync_result": "DONE"
32       },
33       "ats.Job": {
34         "last_sync_finished": "2023-12-29T17:05:45Z",
35         "last_sync_result": "FAILED"
36       }
37     }
38   }
39 }

Create functions for efficiently syncing data

Store the timestamp of when you last started pulling data from Merge as modified_after. Use this timestamp in subsequent API requests to pull updates from Merge since your last sync.

Use the expand parameter to pull multiple models that are related to each other instead of making multiple pulls for related information.

Query parameters:

GET /{endpoint}/?modified_after={timestamp}

Only pull data that has been changed or created since your last sync.

For example, you can ask for modified_after=2021-03-30T20:44:18, and only pull items that are new or changed.

GET /{endpoint}/?expand={common_model_name}

Pull related model information with a single API request.

For example, if you are querying for candidates and also want details about associated applications, you can expand=applications, and Merge will return the actual application objects instead of just the application_id.

Example code snippet

1 from merge_hris_python import EmployeesApi, Configuration, ApiClient
2 from datetime import datetime, timedelta
3 
4 def get_modified_employees(account_token, modified_after, expand):
5     config = Configuration()
6     api_client = ApiClient(configuration=config)
7     employees_api = EmployeesApi(api_client=api_client)
8 
9     # Ensure datetime in ISO 8601 format
10     modified_after = modified_after.isoformat()
11 
12     # Call GET /employees with "modified_after" and "expand" parameters
13     response = employees_api.employees_list(account_token, modified_after=modified_after, expand=expand)
14 
15     return response
16 
17 # Usage
18 account_token = "YOUR_ACCOUNT_TOKEN"
19 modified_after = datetime.now() - timedelta(days=7)  # Get employees modified in the last 7 days
20 expand = "manager,employments"  # Expand manager and employments objects

Sync periodically and poll using /sync-status endpoint

Make sure to implement polling and don’t rely entirely on notification webhooks.

Webhooks can fail for a variety of reasons such as downtime or failed processing.

Merge does attempt to redeliver multiple times using exponential backoff, but we still recommend calling your sync functions periodically every 24 hours.

Make a request to our /sync-status endpoint, which returns an array of syncing statuses for all models in a category. See API reference to learn more.

If status is PARTIALLY SYNCED or DONE, go ahead and retrieve data. If another sync has not started, since the last time you pulled data, there will not be new data.
If status is SYNCING, continue pinging

Example code snippet

1 def is_sync_complete(merge_client, common_model_id):
2     # Get sync statuses for all common models
3     sync_statuses = merge_client.ats.sync_status.list()
4 
5     # Find the status of the specified model
6     sync_status = next(
7         (model for model in sync_statuses.results if model.model_id == common_model_id),
8         None,
9     )
10     if not sync_status:
11         raise ValueError(f"Invalid common model id: {common_model_id}")
12 
13     # Assess the current sync status
14     if sync_status.status == "SYNCING":
15         return False
16 
17     elif sync_status.status in ["FAILED", "DISABLED", "PAUSED"]:
18         raise RuntimeError(f"Sync failed with status: {sync_status.status}")
19 
20     elif sync_status.status in ["DONE", "PARTIALLY_SYNCED"]:
21         return True
22 
23     return False
24 
25 
26 def sync_candidates_data(api_key, account_token, last_synced_from_merge):
27     CANDIDATE_COMMON_MODEL_ID = "ats.Candidate"
28     merge_client = Merge(api_key=api_key, account_token=account_token)
29 
30     # poll for complete sync
31     while not is_sync_complete(merge_client, CANDIDATE_COMMON_MODEL_ID):
32         time.sleep(30)
33 
34     # Retrieve data from the candidates endpoint
35     updated_last_synced_from_merge = timestamp.now()
36     data = merge_client.ats.candidates.list(modified_after=last_synced_from_merge)
37 
38     return data, updated_last_synced_from_merge