Dropwizard Aerospike Tutorial

Dropwizard Aerospike Tutorial

Introduction

        I was searching online for an end to end Dropwizard Aerospike Integration tutorial and I could not find even one. Dropwizard and Aerospike are being used extensively in many startups and other technology giants but there is not a single end to end tutorial online. Hence I decided to come up with this post. This post will give you a comprehensive overview of Dropwizard and Aerospike Integration. The use case that I am trying to address in this post is to call Aerospike from a RESTful Web Service. Aerospike can be used in multiple use cases, however in this post I am using it as a cache on top of a database so that I can retrieve the data that I frequently use, efficiently and much faster than retrieving it from the database. In this post, to keep things simple, I am not using a database instead I am creating and using a HashMap as my database. 

Aerospike

        Aerospike is a high speed, scalable, distributed NoSQL database and key-value store. 

Advantages of Aerospike over other Key-Value, in-memory, NoSQL databases are:
  1. Aerospike does auto-clustering, auto-sharding, auto-rebalancing when the cluster state changes, most of which need manual steps in other databases.
  2. Aerospike is tuned for RAM and SSDs.
  3. Aerospike’s unique storage layer incorporates both DRAM and Flash, giving in-memory characteristics at the price of SSDs. With continual data expiration, Aerospike can operate as a cache with minimal maintenance.
  4. Aerospike's latency is incredibly low, even with high read/write throughput.
You can read more about Aerospike, its usage and its advantages at the Aerospike Website.

        Dropwizard is one of the most popular and most used frameworks for building microservices.
I am assuming that you have a Basic Knowledge of Dropwizard and have a Basic Dropwizard Application running in your machine. If not, please check my blog post on Basic Dropwizard Application by going to the link: Dropwizard Tutorial.

Requirements to Run the Application:
  1. Java
  2. Maven
  3. Aerospike
  4. IDE of your choice
Aerospike should be setup and running in your machine. To setup, run and test if Aerospike is working fine, please refer to my post on: Aerospike Setup.

Once you have a Basic Dropwizard Application and Aerospike up and running in your machine, here are the additional steps required to add the Aerospike integration.

Step 1: Maven Dependencies to be added in pom.xml

Add Aerospike Client to the project by using the following dependency in the pom.xml:
<dependency>
   <groupId>com.aerospike</groupId>
   <artifactId>aerospike-client</artifactId>
   <version>4.0.6</version>
</dependency>
Hence here is the full pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.aj.dropwizardaerospike</groupId>
 <artifactId>DropwizardAerospike</artifactId>
 <version>1.0.0</version>


 <properties>
  <dropwizard.version>1.3.4</dropwizard.version>
        <aerospike.client.version>4.0.6</aerospike.client.version>
        <jdk.version>1.8</jdk.version>
  <packaging>jar</packaging>
 </properties>

 <dependencies>

  <dependency>
            <groupId>io.dropwizard</groupId>
            <artifactId>dropwizard-core</artifactId>
            <version>${dropwizard.version}</version>
        </dependency>

        <dependency>
            <groupId>com.aerospike</groupId>
            <artifactId>aerospike-client</artifactId>
            <version>${aerospike.client.version}</version>
        </dependency>

 </dependencies>

 <build>
  <plugins>
   <plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.0</version>
    <configuration>
     <source>1.8</source>
     <target>1.8</target>
    </configuration>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>2.4</version>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <configuration>
     <createDependencyReducedPom>true</createDependencyReducedPom>
     <filters>
      <filter>
       <artifact>*:*</artifact>
       <excludes>
        <exclude>META-INF/*.SF</exclude>
        <exclude>META-INF/*.DSA</exclude>
        <exclude>META-INF/*.RSA</exclude>
       </excludes>
      </filter>
     </filters>
    </configuration>
    <executions>
     <execution>
      <phase>package</phase>
      <goals>
       <goal>shade</goal>
      </goals>
      <configuration>
       <transformers>
        <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
         <mainClass>com.aj.dropwizardaerospike.DropwizardAerospikeApplication</mainClass>
        </transformer>
       </transformers>
      </configuration>
     </execution>
    </executions>
   </plugin>
  </plugins>
 </build>
</project>

Step 2: Aerospike Managed Object

Most services involve objects which need to be started and stopped: thread pools, database connections, etc. Dropwizard provides the Managed interface for this. Managed interface is a powerful and very useful concept that you should be aware of in Dropwizard. You can either have the class in question implement the start() and stop() methods, or write a wrapper class which does so. Adding a Managed instance to your service’s Environment ties that object’s lifecycle to that of the service’s HTTP server. Before the server starts, the start() method is called. After the server has stopped (and after its graceful shutdown period) the stop() method is called. We will configure Aerospike as a Managed object. The code required to do that is present in AerospikeManaged.java. Here is the code:
package com.aj.dropwizardaerospike;

import com.aerospike.client.AerospikeClient;
import io.dropwizard.lifecycle.Managed;

public class AerospikeManaged implements Managed {

    private AerospikeClient aerospikeClient;

    public AerospikeManaged(AerospikeClient aerospikeClient) {
        this.aerospikeClient = aerospikeClient;
    }

    @Override
    public void start() throws Exception {
    }

    @Override
    public void stop() throws Exception {
        aerospikeClient.close();
    }
}

Step 3: Student POJO

In this post I am using a basic Student POJO as an example. Student has 3 attributes: name, universityId and subjectSpecialization

Here is Student.java:
package com.aj.dropwizardaerospike.domain;

public class Student {

    private String name;
    private String universityId;
    private String subjectSpecialization;

    public Student(String name, String universityId, String subjectSpecialization) {
        this.name = name;
        this.universityId = universityId;
        this.subjectSpecialization = subjectSpecialization;
    }

    public Student() {
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getUniversityId() {
        return universityId;
    }

    public void setUniversityId(String universityId) {
        this.universityId = universityId;
    }

    public String getSubjectSpecialization() {
        return subjectSpecialization;
    }

    public void setSubjectSpecialization(String subjectSpecialization) {
        this.subjectSpecialization = subjectSpecialization;
    }

    @Override
    public String toString() {
        return "Student{" +
                "name='" + name + '\'' +
                ", universityId='" + universityId + '\'' +
                ", subjectSpecialization='" + subjectSpecialization + '\'' +
                '}';
    }
}

Step 4: CacheConfigManager Class

In this class, I am building a new Student Cache where I first try to fetch the student data by key from the cache. If it is not present in the cache it will fetch the data from the database and populate the cache. Here is the class: CacheConfigManager.java
package com.aj.dropwizardaerospike.cache;

import com.aerospike.client.AerospikeClient;
import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration;
import com.aj.dropwizardaerospike.domain.Student;
import com.aj.dropwizardaerospike.service.StudentService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CacheConfigManager {

    private static final Logger logger = LoggerFactory.getLogger(CacheConfigManager.class);

    private static CacheConfigManager cacheConfigManager = new CacheConfigManager();

    public static CacheConfigManager getInstance() {
        return cacheConfigManager;
    }

    //Logic For Student Cache
    public Student getStudentData(String key, StudentService studentService,
                                  AerospikeClient aerospikeClient, DropwizardAerospikeConfiguration config) {
        try {
            Student student = studentService.getFromCache(aerospikeClient,key,
                    config.getNamespace(), config.getSetName());
            if(student == null){
                student = studentService.getFromDatabase(key);
                studentService.createStudent(aerospikeClient, student.getUniversityId(),
                        student.getName(), student.getSubjectSpecialization(),
                        config.getNamespace(), config.getSetName());
            }
            return student;
        } catch (Exception e) {
            logger.error("Error Retrieving Elements from the Student Cache"
                    + e.getMessage());
            return null;
        }
    }
}

Step 5: Student Service Class

To keep things simple, I am not using a database instead I am creating and using a HashMap as my database. This HashMap has universityId as key and the Student Object as Value. In this class, I have the following methods:

1. getFromDatabase, which will be called whenever the cache does not have the key that I am looking for.

2. createStudent, which creates a student record in Aerospike.
There are a few important concepts to understand in creating a new record in Aerospike, that are being used in the method createStudent:
a. WritePolicy: WritePolicy is a container object for policy attributes used in write operations. This object is used in methods where database writes can occur. If you observe the code below you can see how I have created a new WritePolicy. I have set 2 attributes on the WritePolicy:
recordExistsAction: This defines how to handle writes where the record already exists.
expiration: This defines record expiration time in seconds. I have used 300 seconds.
To know more about Class WritePolicy, please check: Class WritePolicy.
b. Key: A key uniquely identifies a record in the Aerospike database within a given namespace. New key is created using 3 parameters: namespace, set and key. To know more about Class Key, please check: Class Key. To see its usage, please check the code below.
c. Bin: A bin is a Column name/value pair. To know more about Class Bin, please check: Class Bin. To see its usage, please check the code below.
The Aerospike Java client library provides AerospikeClient.put() for writing records in the database. Note that if they do not already exist, set and bins are automatically created. You don't have to define a schema for the database. To see its usage, please check the code below.

3. getFromCache, which fetches the student record from Aerospike and returns a Student POJO. AerospikeClient.get() method returns a Record object that contains the metadata and bins of the record. To see its usage, please check the code below in the method getFromCache.

Here is StudentService.java:
package com.aj.dropwizardaerospike.service;

import com.aerospike.client.*;
import com.aerospike.client.policy.RecordExistsAction;
import com.aerospike.client.policy.WritePolicy;
import com.aj.dropwizardaerospike.domain.Student;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;
import java.util.Map;

public class StudentService {

    private static final Logger logger = LoggerFactory.getLogger(StudentService.class);

    public StudentService() {
    }

    public static Student getFromDatabase(String universityId) {
        Student student1 = new Student("Jim", "S100", "Science");
        Student student2 = new Student("Steve", "M101", "Maths");
        Student student3 = new Student("Mark", "P102", "Physics");
        Map<String, Student> database = new HashMap<>();
        database.put("S100", student1);
        database.put("M101", student2);
        database.put("P102", student3);
        logger.info("Database called for: {}", universityId);
        return database.get(universityId);
    }

    public void createStudent(AerospikeClient client, String universityId,
                              String name, String subjectSpecialization,
                              String namespace, String setName) throws AerospikeException {
        // Write record
        WritePolicy wPolicy = new WritePolicy();
        wPolicy.recordExistsAction = RecordExistsAction.UPDATE;
        //Cache will expire after 300 Seconds(5 minutes)
        wPolicy.expiration = 300;

        Key key = new Key(namespace, setName, universityId);
        Bin bin1 = new Bin("name", name);
        Bin bin2 = new Bin("universityId", universityId);
        Bin bin3 = new Bin("specialization", subjectSpecialization);
        client.put(wPolicy, key, bin1, bin2, bin3);
        logger.info("Student record created in cache with universityId: {}", universityId);
    }

    public Student getFromCache(AerospikeClient client, String universityId,
                                String namespace, String setName) throws AerospikeException {
        Record studentRecord;
        Key studentKey;
        Student student = null;
        studentKey = new Key(namespace, setName, universityId);
        studentRecord = client.get(null, studentKey);
        if (studentRecord != null) {
            student = new Student();
            student.setName(studentRecord.getValue("name").toString());
            student.setUniversityId(studentRecord.getValue("universityId").toString());
            student.setSubjectSpecialization(studentRecord.getValue("specialization").toString());
            logger.info("Cache called for: {}", universityId);
        }
        return student;
    }
}

Step 6: API to test Cache

I have created a GET API in Dropwizard which will call method to get student data from cache. I have written code to call this method 3 times. So during the invocation, if you check the application logs, it will clearly show that the data is retrieved from the database during the first call. During the first call the cache will also get populated. Hence the subsequent calls will retrieve data from the Cache.

Here is StudentResource.java:
package com.aj.dropwizardaerospike.resource;

import com.aerospike.client.AerospikeClient;
import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration;
import com.aj.dropwizardaerospike.cache.CacheConfigManager;
import com.aj.dropwizardaerospike.service.StudentService;
import com.codahale.metrics.annotation.Timed;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.HashMap;
import java.util.Map;

@Path("/student")
@Produces(MediaType.APPLICATION_JSON)
public class StudentResource {

    private static final Logger logger = LoggerFactory.getLogger(StudentResource.class);

    private static StudentService studentService;
    private static AerospikeClient aerospikeClient;
    private static DropwizardAerospikeConfiguration config;

    public StudentResource(StudentService studentService, AerospikeClient aerospikeClient,
                           DropwizardAerospikeConfiguration config) {
        this.studentService = studentService;
        this.aerospikeClient = aerospikeClient;
        this.config = config;
    }

    @Timed
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response cache() {
        logger.info("In StudentResource.cache()...Get Student Data");
        //On the first call, data will be fetched from DB and
        //cache will be populated with the corresponding student record
        //On all subsequent calls, data will be returned from the cache
        for (int i = 1; i < 4; i++) {
            getStudentData(i);
        }
        Map<String, String> response = new HashMap<>();
        response.put("message", "Student Data has been retrieved");
        return Response.ok(response).build();
    }

    private void getStudentData(int i) {
        logger.info("********** Call " + String.valueOf(i) + " Started **********");
        logger.info("Call " + String.valueOf(i) + ": {}",
                CacheConfigManager.getInstance().getStudentData("S100",studentService,
                        aerospikeClient, config));
        logger.info("Call " + String.valueOf(i) + ": {}",
                CacheConfigManager.getInstance().getStudentData("M101",studentService,
                        aerospikeClient, config));
        logger.info("Call " + String.valueOf(i) + ": {}",
                CacheConfigManager.getInstance().getStudentData("P102",studentService,
                        aerospikeClient, config ));
        logger.info("********** Call " + String.valueOf(i) + " Ended **********");
    }
}

Step 7: Configuration in yml file and in the Configuration class

In the yml file, I am setting the default namespace in Aerospike which is test and Set Name as: student. I am also setting the Application Name as DropwizardAerospike, Application server port as 4000 and adminContextPath as: /admin
Here is the full yml file dropwizardaerospike.yml:
logging:
  level: INFO
  
  appenders:
    - type: console
      threshold: ALL
      timeZone: IST
      
server:
  type: simple
  applicationContextPath: /
  adminContextPath: /admin
  connector:
      port: 4000
      type: http

appName : DropwizardAerospike

namespace: test
setName: student
Getters and setters for the properties in the yml file are created in the Configuration class.
Here is the file: DropwizardAerospikeConfiguration.java:
package com.aj.dropwizardaerospike;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.dropwizard.Configuration;

@JsonIgnoreProperties(ignoreUnknown = true)
public class DropwizardAerospikeConfiguration extends Configuration {

    @JsonProperty
    public String appName;

    @JsonProperty
    public String namespace;

    @JsonProperty
    public String setName;

    public String getAppName() {
        return appName;
    }

    public void setAppName(String appName) {
        this.appName = appName;
    }

    public String getNamespace() {
        return namespace;
    }

    public void setNamespace(String namespace) {
        this.namespace = namespace;
    }

    public String getSetName() {
        return setName;
    }

    public void setSetName(String setName) {
        this.setName = setName;
    }
}

Step 8: Create Health Check Class 

In the Health Check Resource, I am performing a basic check of the Application Name retrieved from the file: dropwizardaerospike.yml
Here is the file: DropwizardAerospikeHealthCheckResource.java:
package com.aj.dropwizardaerospike.resource;

import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration;
import com.codahale.metrics.health.HealthCheck;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class DropwizardAerospikeHealthCheckResource extends HealthCheck {

    private static final Logger logger = LoggerFactory.getLogger(DropwizardAerospikeHealthCheckResource.class);

    private static String appName;

    public DropwizardAerospikeHealthCheckResource(DropwizardAerospikeConfiguration dropwizardAerospikeConfiguration){
       this.appName = dropwizardAerospikeConfiguration.getAppName();
    }

    @Override
    protected Result check() throws Exception {
        logger.info("App Name is: {}", appName);
        if("DropwizardAerospike".equalsIgnoreCase(appName)) {
            return Result.healthy();
        }
        return Result.unhealthy("DropwizardAerospike Service is down");
    }
}

Step 9: Create the Application Class 

Everything comes together in the application class. We register the resources, health check and Aerospike Managed Object in the application class. 
Important things to observe in the Application Class:
1. Creation of AerospikeClient: In the code below, I have used the host name (172.28.128.3) as shown in the Oracle VM VirtualBox (see image below) and port 3000. For instructions on how to use Aerospike host using Oracle VM VirtualBox Manager, please refer to my post on: Aerospike Setup
Here is the file: DropwizardAerospikeApplication.java
package com.aj.dropwizardaerospike;

import com.aerospike.client.AerospikeClient;
import com.aj.dropwizardaerospike.resource.DropwizardAerospikeHealthCheckResource;
import com.aj.dropwizardaerospike.resource.PingResource;
import com.aj.dropwizardaerospike.resource.StudentResource;
import com.aj.dropwizardaerospike.service.StudentService;
import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class DropwizardAerospikeApplication extends Application<DropwizardAerospikeConfiguration> {

    private static final Logger logger = LoggerFactory.getLogger(DropwizardAerospikeApplication.class);

 public static void main(String[] args) throws Exception {
  new DropwizardAerospikeApplication().run("server", args[0]);
 }

    @Override
    public void initialize(Bootstrap<DropwizardAerospikeConfiguration> b) {
    }

 @Override
 public void run(DropwizardAerospikeConfiguration config, Environment env)
   throws Exception {
  AerospikeClient aerospikeClient = new AerospikeClient("172.28.128.3", 3000);
     AerospikeManaged aerospikeManaged = new AerospikeManaged(aerospikeClient);
        env.lifecycle().manage(aerospikeManaged);
     StudentService studentService = new StudentService();
     logger.info("Registering RESTful API resources");
  env.jersey().register(new PingResource());
        env.jersey().register(new StudentResource(studentService, aerospikeClient, config));
  env.healthChecks().register("DropwizardAerospikeHealthCheck",
    new DropwizardAerospikeHealthCheckResource(config));
 }
}

Project Setup:

For Java Setup, please refer to:  Java Setup
For Maven Setup, please refer to: Maven Setup
For Git and Project Setup, please refer to: Git and Project Setup

Run Application:

1. To run application in your IDE use:
    Program arguments: src/main/resources/dropwizardaerospike.yml
 
2. To run JAR from command prompt:
     Build jar by using command:   
   mvn clean install
     Run JAR by using command in Project folder location:
   java -jar target\DropwizardAerospike-1.0.0.jar src\main\resources\dropwizardaerospike.yml

API calls and results:

1. GET API to test cache
    http://localhost:4000/student
When this API is called, if you check the logs, you can see that in the first call, as data is not present in cache, it is retrieved from the Database (HashMap). Also the cache gets populated with the data in Call 1. So in Call 2 and Call 3 you can see that data is being retrieved from the cache.
When the API is executed, you can check the data populated in the set in Aerospike using aql (Aerospike Query Language). For this, login to the Aerospike host. For instructions on how to login to Aerospike host using Oracle VM VirtualBox Manager, please refer to my post on: Aerospike Setup
Enter the following command to start using aql on the host:
aql -T 10000
Query the set student, that we created in the application above, using:
select * from test.student
Other useful aql queries:
show namespaces
show sets
show bins
You can also check the reads and writes in Aerospike using the Aerospike Management Console (AMC). For instructions on how to access AMC, please refer to my post on: Aerospike Setup
Other than these APIs, this application has the following APIs:

1. GET API for Application Health Check:
http://localhost:4000/admin/healthcheck
2. GET API to Ping and test if the application is up and running: 
    http://localhost:4000/ping
3. POST API to Ping and test if the application is up and running: 
    http://localhost:4000/ping
    JSON Request Body:
    {
     "input": "ping"
    }
4. GET Admin API to see application metrics: Dropwizard provides this Admin API. As I have set adminContextPath as: /admin in dropwizardmongodb.yml the link I need to use in any browser is: http://localhost:4000/admin/
You can click on individual links on this page to see different application metrics.

Conclusion and GitHub link:

    In this post I have shown you how you can integrate your existing Dropwizard Application with Aerospike and perform basic operations on Aerospike. The code used in this post is available on GitHub.
    Learn the most popular and trending technologies like Machine Learning, Chatbots, Angular 5, Internet of Things (IoT), Akka HTTP, Play Framework, Dropwizard, Docker, Elastic Stack, Netflix Eureka, Netflix Zuul, Spring Cloud, Spring Boot, Flask and RESTful Web Service integration with MongoDB, Kafka, Redis, Aerospike, MySQL DB in simple steps by reading my most popular blog posts at Software Developer Central.
    If you like my post, please feel free to share it using the share button just below this paragraph or next to the heading of the post. You can also tweet with #SoftwareDeveloperCentral on Twitter. To get a notification on my latest posts or to keep the conversation going, you can follow me on Twitter or Instagram. Please leave a note below if you have any questions or comments.

Comments

Popular Posts

Golang gRPC Microservice

Dropwizard MySQL Integration Tutorial

Asynchronous Processing (@Async) in Spring Boot