Asynchronous Processing

SAPI-NT supports asynchronous (batch) processing of requests. This is intended to help the API to respond to long-running and high-volume requests by using the following measures:

• Requests are queued and executed sequentially, not concurrently.
• SAPI-NT can request privileged access to the necessary resources (for example by requesting an extended query timeout for a Fuseki database).
• The formatted results of the request are cached so that they can be retrieved immediately on subsequent requests.

The queue is typically a data store such as a SQL database, but could also be an in-memory store.

The cache is typically a local or remote file system such as Amazon S3 where results can be stored in an appropriate file format.

The underlying framework is armlib-spring, which is derived from armlib by adding Kotlin and Spring framework integration.

SAPI-NT does not control the maintenance or clearing of the queue or cache. If you want to invalidate or add pre-made entries to the cache, you can do so independently of your SAPI-NT application.

Basic Life Cycle

1. The API author configures the queue and cache implementation for their SAPI-NT application, using standard Spring Beans functionality.
2. On startup, the SAPI-NT application connects to the queue and begins polling for pending requests.
3. On receiving a batch request, it is serialized and stored in the queue as a pending request. The caller receives a response to let them know that the request is pending.
4. When the pending request is detected, the application starts to execute it and marks it as in progress.
5. When the request is finished, it is marked as complete or failed. If it was completed successfully, the results of the request are stored as a file in the cache.
6. When the same batch request is subsequently received again, the application will return the contents of the cached file as its response.

Configuration

API

Batch processing is configured at the API level by certain Spring application properties that are stored under the sapi-nt.batch namespace.

The following properties determine how the API will respond to batch requests.

The following properties determine how batch requests that are queued should be processed.

The most important properties are sapi-nt.batch.enabled and sapi-nt.batch.process.enabled, which must both be set to true in order to fully enable the batch processing features of SAPI-NT.

However, you may want to use a different application to process queued requests while making the results accessible through your SAPI-NT API. In this case, you should set the sapi-nt.batch.enabled property to true and the sapi-nt.batch.process.enabled property to false.

Endpoints

With batch processing enabled at the API level, you can configure whether it will be available for each of your endpoints. This is determined by the following properties in the endpoint specification:

The data source that you specify should be configured to sustain long-running and high-volume queries, for example by having an increased time limit. The underlying store may be the same as the usual data source for the endpoint.

Note that if you have configured a softLimit for your endpoint that limits the number of items retrieved when no user limit is given, it will be ignored when executing the query in batch mode.

Queue and Cache

The queue is a data store which keeps track of requests and their status. SAPI-NT interacts with the store through the QueueManager interface defined in armlib-spring.

The cache is a file system which stores the results of previously completed requests as files. SAPI-NT interacts with the cache through the CacheManager interface defined in armlib-spring.

You can use one of the standard queue and cache manager implementations that are included in armlib-spring-impl or write a custom implementation. If you want to use a standard implementation, you must add the armlib-spring-impl dependency to your project POM.

<dependency>
    <groupId>com.epimorphics</groupId>
    <artifactId>armlib-spring-impl</artifactId>
    <version>0.2.2</version>
</dependency>

To configure the queue and cache managers, and to make them available to the SAPI-NT application context, define them as Spring Beans by using Spring annotations or XML configuration. The exact configuration options will depend on the implementations you choose.

Note

If you use XML configuration, you must make sure that your application imports the corresponding resource. You can do this by adding the @ImportResource("classpath:your-file-here.xml") annotation to your main class (usually SapiNtApplication).

Example XML Configuration

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
			 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
			 xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

	<bean id="cacheManager" class="com.epimorphics.armlib.impl.S3CacheManager">
		<property name="compressed" value="true" />
	</bean>

	<bean id="queueManager" class="com.epimorphics.armlib.impl.SqlQueueManager">
		<constructor-arg name="config" ref="queueManagerConfig"/>
	</bean>

	<bean id="queueManagerConfig" class="com.epimorphics.armlib.impl.SqlQueueManager.Config">
		<constructor-arg name="deleteOnComplete" value="false"/>
	</bean>
</beans>

Spring Auto Configuration

Sapi-nt and armlib make use of Spring auto-configuration to initialise certain beans, such as DataSource, if needed. This allows any implementation to be injected into the application automatically simply by adding the dependency to the classpath. However, this may result in errors at start-up if the auto-configured beans are misconfigured for your environment. In that case, you should use profile-based configuration to define your intended usage.

Data Source

When using the SQL queue manager implementation, you will need to configure a data source. If you do not, and auto-configuration is enabled, your application will emit the following error on start-up:

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class

You can prevent this error by either disabling auto-configuration of data sources (see the section on profile-based configuration), or by configuring the spring.datasource.* properties in your application.yaml file. The basic properties you will need to initialise your data source are the following:

Read the Spring documentation for additional options.

Example

spring:
  datasource:
    driverClassName: org.postgresql.Driver
    url: jdbc:postgresql://localhost:5432/db

Profile Based Configuration

You may need to use distinct configurations for production and development environments. You can do this by using Spring profiles and defining profile-specific config options and Spring beans.

For profile-based YAML configuration, create a file named application-{profile}.yaml in the resources directory of your application. For example, to disable the auto-configuration of the SQL data source in development, create the following application-dev.yaml:

spring:
  autoconfigure:
    exclude:
    - "org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration"

For profile-based XML bean configuration, create a separate beans element with a profile attribute for each of your profiles. For example, to use S3 and SQL cache and queue implementations by default, and file and memory implementations in development, change your XML configuration to the following:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
			 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
			 xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

	<beans profile="default">
		<bean id="cacheManager" class="com.epimorphics.armlib.impl.S3CacheManager">
			<property name="compressed" value="true" />
		</bean>

		<bean id="queueManager" class="com.epimorphics.armlib.impl.SqlQueueManager">
			<constructor-arg name="config" ref="queueManagerConfig"/>
		</bean>

		<bean id="queueManagerConfig" class="com.epimorphics.armlib.impl.SqlQueueManager.Config">
			<constructor-arg name="deleteOnComplete" value="false"/>
		</bean>
	</beans>

	<beans profile="dev">
		<bean id="cacheManager" class="com.epimorphics.armlib.impl.FileCacheManager">
		</bean>

		<bean id="queueManager" class="com.epimorphics.armlib.impl.MemQueueManager">
		</bean>
	</beans>
</beans>

To run the sapi-nt application under a certain profile, you can use the --spring.profiles.active={profile} program argument. For example, to use the dev profile, supply the argument --spring.profiles.active=dev.

Usage

See the API documentation for information about how to use the batch API.