Dropwizard Aerospike Tutorial
Dropwizard Aerospike Tutorial
Introduction
I was searching online for an end to end
Dropwizard Aerospike Integration tutorial and I could not find even one.
Dropwizard and Aerospike are being used extensively in many startups and
other technology giants but there is not a single end to end tutorial
online. Hence I decided to come up with this post. This post will give you a
comprehensive overview of Dropwizard and Aerospike Integration. The use case
that I am trying to address in this post is to call Aerospike from a RESTful
Web Service. Aerospike can be used in multiple use cases, however in this
post I am using it as a cache on top of a database so that I can retrieve
the data that I frequently use, efficiently and much faster than retrieving
it from the database. In this post, to keep things simple, I am not using a
database instead I am creating and using a HashMap as my database.
Aerospike
Aerospike is a high speed, scalable, distributed
NoSQL database and key-value store.
Advantages of Aerospike over other Key-Value, in-memory, NoSQL databases
are:
- Aerospike does auto-clustering, auto-sharding, auto-rebalancing when the cluster state changes, most of which need manual steps in other databases.
- Aerospike is tuned for RAM and SSDs.
- Aerospike’s unique storage layer incorporates both DRAM and Flash, giving in-memory characteristics at the price of SSDs. With continual data expiration, Aerospike can operate as a cache with minimal maintenance.
- Aerospike's latency is incredibly low, even with high read/write throughput.
You can read more about Aerospike, its usage and its advantages at the
Aerospike Website.
Dropwizard is one of the most popular and most
used frameworks for building microservices.
I am assuming that you have a Basic Knowledge of Dropwizard and have a Basic
Dropwizard Application running in your machine. If not, please check my blog
post on Basic Dropwizard Application by going to the link: Dropwizard Tutorial.
Requirements to Run the Application:
Once you have a Basic Dropwizard Application and Aerospike up and running in your machine, here are the additional steps required to add the Aerospike integration.
Requirements to Run the Application:
- Java
- Maven
- Aerospike
- IDE of your choice
Once you have a Basic Dropwizard Application and Aerospike up and running in your machine, here are the additional steps required to add the Aerospike integration.
Step 1: Maven Dependencies to be added in pom.xml
Add Aerospike Client to the project by using the following dependency in the
pom.xml:
<dependency> <groupId>com.aerospike</groupId> <artifactId>aerospike-client</artifactId> <version>4.0.6</version> </dependency>Hence here is the full pom.xml:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.aj.dropwizardaerospike</groupId> <artifactId>DropwizardAerospike</artifactId> <version>1.0.0</version> <properties> <dropwizard.version>1.3.4</dropwizard.version> <aerospike.client.version>4.0.6</aerospike.client.version> <jdk.version>1.8</jdk.version> <packaging>jar</packaging> </properties> <dependencies> <dependency> <groupId>io.dropwizard</groupId> <artifactId>dropwizard-core</artifactId> <version>${dropwizard.version}</version> </dependency> <dependency> <groupId>com.aerospike</groupId> <artifactId>aerospike-client</artifactId> <version>${aerospike.client.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.0</version> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <version>2.4</version> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <configuration> <createDependencyReducedPom>true</createDependencyReducedPom> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>com.aj.dropwizardaerospike.DropwizardAerospikeApplication</mainClass> </transformer> </transformers> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>
Step 2: Aerospike Managed Object
Most services involve objects which need to be started and stopped: thread
pools, database connections, etc. Dropwizard provides
the Managed interface for this. Managed interface is a
powerful and very useful concept that you should be aware of in Dropwizard.
You can either have the class in question implement
the start() and stop() methods, or write a
wrapper class which does so. Adding a Managed instance to
your service’s Environment ties that object’s lifecycle to
that of the service’s HTTP server. Before the server starts,
the start() method is called. After the server has stopped
(and after its graceful shutdown period) the stop() method
is called. We will configure Aerospike as a Managed object.
The code required to do that is present
in AerospikeManaged.java. Here is the code:
package com.aj.dropwizardaerospike; import com.aerospike.client.AerospikeClient; import io.dropwizard.lifecycle.Managed; public class AerospikeManaged implements Managed { private AerospikeClient aerospikeClient; public AerospikeManaged(AerospikeClient aerospikeClient) { this.aerospikeClient = aerospikeClient; } @Override public void start() throws Exception { } @Override public void stop() throws Exception { aerospikeClient.close(); } }
Step 3: Student POJO
In this post I am using a basic Student POJO as an example. Student has 3
attributes: name, universityId and subjectSpecialization
Here is Student.java:
Here is Student.java:
package com.aj.dropwizardaerospike.domain; public class Student { private String name; private String universityId; private String subjectSpecialization; public Student(String name, String universityId, String subjectSpecialization) { this.name = name; this.universityId = universityId; this.subjectSpecialization = subjectSpecialization; } public Student() { } public String getName() { return name; } public void setName(String name) { this.name = name; } public String getUniversityId() { return universityId; } public void setUniversityId(String universityId) { this.universityId = universityId; } public String getSubjectSpecialization() { return subjectSpecialization; } public void setSubjectSpecialization(String subjectSpecialization) { this.subjectSpecialization = subjectSpecialization; } @Override public String toString() { return "Student{" + "name='" + name + '\'' + ", universityId='" + universityId + '\'' + ", subjectSpecialization='" + subjectSpecialization + '\'' + '}'; } }
Step 4: CacheConfigManager Class
In this class, I am building a new Student Cache where I first try to fetch
the student data by key from the cache. If it is not present in the cache it
will fetch the data from the database and populate the cache. Here is the
class: CacheConfigManager.java
package com.aj.dropwizardaerospike.cache; import com.aerospike.client.AerospikeClient; import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration; import com.aj.dropwizardaerospike.domain.Student; import com.aj.dropwizardaerospike.service.StudentService; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class CacheConfigManager { private static final Logger logger = LoggerFactory.getLogger(CacheConfigManager.class); private static CacheConfigManager cacheConfigManager = new CacheConfigManager(); public static CacheConfigManager getInstance() { return cacheConfigManager; } //Logic For Student Cache public Student getStudentData(String key, StudentService studentService, AerospikeClient aerospikeClient, DropwizardAerospikeConfiguration config) { try { Student student = studentService.getFromCache(aerospikeClient,key, config.getNamespace(), config.getSetName()); if(student == null){ student = studentService.getFromDatabase(key); studentService.createStudent(aerospikeClient, student.getUniversityId(), student.getName(), student.getSubjectSpecialization(), config.getNamespace(), config.getSetName()); } return student; } catch (Exception e) { logger.error("Error Retrieving Elements from the Student Cache" + e.getMessage()); return null; } } }
Step 5: Student Service Class
To keep things simple, I am not using a database instead I am creating and
using a HashMap as my database. This HashMap has universityId as key and the
Student Object as Value. In this class, I have the following methods:
1. getFromDatabase, which will be called whenever
the cache does not have the key that I am looking for.
2. createStudent, which creates a student record in
Aerospike.
There are a few important concepts to understand in creating a new record in Aerospike, that are being used in the method createStudent:
a. WritePolicy: WritePolicy is a container object for policy attributes used in write operations. This object is used in methods where database writes can occur. If you observe the code below you can see how I have created a new WritePolicy. I have set 2 attributes on the WritePolicy:
recordExistsAction: This defines how to handle writes where the record already exists.
expiration: This defines record expiration time in seconds. I have used 300 seconds.
To know more about Class WritePolicy, please check: Class WritePolicy.
b. Key: A key uniquely identifies a record in the Aerospike database within a given namespace. New key is created using 3 parameters: namespace, set and key. To know more about Class Key, please check: Class Key. To see its usage, please check the code below.
c. Bin: A bin is a Column name/value pair. To know more about Class Bin, please check: Class Bin. To see its usage, please check the code below.
The Aerospike Java client library provides AerospikeClient.put() for writing records in the database. Note that if they do not already exist, set and bins are automatically created. You don't have to define a schema for the database. To see its usage, please check the code below.
There are a few important concepts to understand in creating a new record in Aerospike, that are being used in the method createStudent:
a. WritePolicy: WritePolicy is a container object for policy attributes used in write operations. This object is used in methods where database writes can occur. If you observe the code below you can see how I have created a new WritePolicy. I have set 2 attributes on the WritePolicy:
recordExistsAction: This defines how to handle writes where the record already exists.
expiration: This defines record expiration time in seconds. I have used 300 seconds.
To know more about Class WritePolicy, please check: Class WritePolicy.
b. Key: A key uniquely identifies a record in the Aerospike database within a given namespace. New key is created using 3 parameters: namespace, set and key. To know more about Class Key, please check: Class Key. To see its usage, please check the code below.
c. Bin: A bin is a Column name/value pair. To know more about Class Bin, please check: Class Bin. To see its usage, please check the code below.
The Aerospike Java client library provides AerospikeClient.put() for writing records in the database. Note that if they do not already exist, set and bins are automatically created. You don't have to define a schema for the database. To see its usage, please check the code below.
3. getFromCache, which fetches the student record from Aerospike and
returns a Student POJO. AerospikeClient.get() method returns a Record object
that contains the metadata and bins of the record. To see its usage, please
check the code below in the method getFromCache.
Here is StudentService.java:
package com.aj.dropwizardaerospike.service; import com.aerospike.client.*; import com.aerospike.client.policy.RecordExistsAction; import com.aerospike.client.policy.WritePolicy; import com.aj.dropwizardaerospike.domain.Student; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.HashMap; import java.util.Map; public class StudentService { private static final Logger logger = LoggerFactory.getLogger(StudentService.class); public StudentService() { } public static Student getFromDatabase(String universityId) { Student student1 = new Student("Jim", "S100", "Science"); Student student2 = new Student("Steve", "M101", "Maths"); Student student3 = new Student("Mark", "P102", "Physics"); Map<String, Student> database = new HashMap<>(); database.put("S100", student1); database.put("M101", student2); database.put("P102", student3); logger.info("Database called for: {}", universityId); return database.get(universityId); } public void createStudent(AerospikeClient client, String universityId, String name, String subjectSpecialization, String namespace, String setName) throws AerospikeException { // Write record WritePolicy wPolicy = new WritePolicy(); wPolicy.recordExistsAction = RecordExistsAction.UPDATE; //Cache will expire after 300 Seconds(5 minutes) wPolicy.expiration = 300; Key key = new Key(namespace, setName, universityId); Bin bin1 = new Bin("name", name); Bin bin2 = new Bin("universityId", universityId); Bin bin3 = new Bin("specialization", subjectSpecialization); client.put(wPolicy, key, bin1, bin2, bin3); logger.info("Student record created in cache with universityId: {}", universityId); } public Student getFromCache(AerospikeClient client, String universityId, String namespace, String setName) throws AerospikeException { Record studentRecord; Key studentKey; Student student = null; studentKey = new Key(namespace, setName, universityId); studentRecord = client.get(null, studentKey); if (studentRecord != null) { student = new Student(); student.setName(studentRecord.getValue("name").toString()); student.setUniversityId(studentRecord.getValue("universityId").toString()); student.setSubjectSpecialization(studentRecord.getValue("specialization").toString()); logger.info("Cache called for: {}", universityId); } return student; } }
Step 6: API to test Cache
I have created a GET API in Dropwizard which will call method to get
student data from cache. I have written code to call this method 3 times.
So during the invocation, if you check the application logs, it will
clearly show that the data is retrieved from the database during the first
call. During the first call the cache will also get populated. Hence the
subsequent calls will retrieve data from the Cache.
Here is StudentResource.java:
package com.aj.dropwizardaerospike.resource; import com.aerospike.client.AerospikeClient; import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration; import com.aj.dropwizardaerospike.cache.CacheConfigManager; import com.aj.dropwizardaerospike.service.StudentService; import com.codahale.metrics.annotation.Timed; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; import java.util.HashMap; import java.util.Map; @Path("/student") @Produces(MediaType.APPLICATION_JSON) public class StudentResource { private static final Logger logger = LoggerFactory.getLogger(StudentResource.class); private static StudentService studentService; private static AerospikeClient aerospikeClient; private static DropwizardAerospikeConfiguration config; public StudentResource(StudentService studentService, AerospikeClient aerospikeClient, DropwizardAerospikeConfiguration config) { this.studentService = studentService; this.aerospikeClient = aerospikeClient; this.config = config; } @Timed @GET @Produces(MediaType.APPLICATION_JSON) public Response cache() { logger.info("In StudentResource.cache()...Get Student Data"); //On the first call, data will be fetched from DB and //cache will be populated with the corresponding student record //On all subsequent calls, data will be returned from the cache for (int i = 1; i < 4; i++) { getStudentData(i); } Map<String, String> response = new HashMap<>(); response.put("message", "Student Data has been retrieved"); return Response.ok(response).build(); } private void getStudentData(int i) { logger.info("********** Call " + String.valueOf(i) + " Started **********"); logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentData("S100",studentService, aerospikeClient, config)); logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentData("M101",studentService, aerospikeClient, config)); logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentData("P102",studentService, aerospikeClient, config )); logger.info("********** Call " + String.valueOf(i) + " Ended **********"); } }
Step 7: Configuration in yml file and in the Configuration class
In the yml file, I am setting the default namespace in Aerospike
which is test and Set Name as: student. I
am also setting the Application Name as DropwizardAerospike,
Application server port as 4000 and adminContextPath
as: /admin
Here is the full yml file dropwizardaerospike.yml:
logging: level: INFO appenders: - type: console threshold: ALL timeZone: IST server: type: simple applicationContextPath: / adminContextPath: /admin connector: port: 4000 type: http appName : DropwizardAerospike namespace: test setName: studentGetters and setters for the properties in the yml file are created in the Configuration class.
Here is the file: DropwizardAerospikeConfiguration.java:
package com.aj.dropwizardaerospike; import com.fasterxml.jackson.annotation.JsonIgnoreProperties; import com.fasterxml.jackson.annotation.JsonProperty; import io.dropwizard.Configuration; @JsonIgnoreProperties(ignoreUnknown = true) public class DropwizardAerospikeConfiguration extends Configuration { @JsonProperty public String appName; @JsonProperty public String namespace; @JsonProperty public String setName; public String getAppName() { return appName; } public void setAppName(String appName) { this.appName = appName; } public String getNamespace() { return namespace; } public void setNamespace(String namespace) { this.namespace = namespace; } public String getSetName() { return setName; } public void setSetName(String setName) { this.setName = setName; } }
Step 8: Create Health Check Class
In the Health Check Resource, I am performing a basic check of the
Application Name retrieved from the
file: dropwizardaerospike.yml.
Here is the
file: DropwizardAerospikeHealthCheckResource.java:
package com.aj.dropwizardaerospike.resource; import com.aj.dropwizardaerospike.DropwizardAerospikeConfiguration; import com.codahale.metrics.health.HealthCheck; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class DropwizardAerospikeHealthCheckResource extends HealthCheck { private static final Logger logger = LoggerFactory.getLogger(DropwizardAerospikeHealthCheckResource.class); private static String appName; public DropwizardAerospikeHealthCheckResource(DropwizardAerospikeConfiguration dropwizardAerospikeConfiguration){ this.appName = dropwizardAerospikeConfiguration.getAppName(); } @Override protected Result check() throws Exception { logger.info("App Name is: {}", appName); if("DropwizardAerospike".equalsIgnoreCase(appName)) { return Result.healthy(); } return Result.unhealthy("DropwizardAerospike Service is down"); } }
Step 9: Create the Application Class
Everything comes together in the application class. We register the
resources, health check and Aerospike Managed Object in the application
class.
Important things to observe in the Application Class:
1. Creation of AerospikeClient: In the code below, I have used
the host name (172.28.128.3) as shown in the Oracle VM VirtualBox (see image
below) and port 3000. For instructions on how to use Aerospike host using
Oracle VM VirtualBox Manager, please refer to my post on: Aerospike Setup
Here is the file: DropwizardAerospikeApplication.java:
package com.aj.dropwizardaerospike; import com.aerospike.client.AerospikeClient; import com.aj.dropwizardaerospike.resource.DropwizardAerospikeHealthCheckResource; import com.aj.dropwizardaerospike.resource.PingResource; import com.aj.dropwizardaerospike.resource.StudentResource; import com.aj.dropwizardaerospike.service.StudentService; import io.dropwizard.Application; import io.dropwizard.setup.Bootstrap; import io.dropwizard.setup.Environment; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class DropwizardAerospikeApplication extends Application<DropwizardAerospikeConfiguration> { private static final Logger logger = LoggerFactory.getLogger(DropwizardAerospikeApplication.class); public static void main(String[] args) throws Exception { new DropwizardAerospikeApplication().run("server", args[0]); } @Override public void initialize(Bootstrap<DropwizardAerospikeConfiguration> b) { } @Override public void run(DropwizardAerospikeConfiguration config, Environment env) throws Exception { AerospikeClient aerospikeClient = new AerospikeClient("172.28.128.3", 3000); AerospikeManaged aerospikeManaged = new AerospikeManaged(aerospikeClient); env.lifecycle().manage(aerospikeManaged); StudentService studentService = new StudentService(); logger.info("Registering RESTful API resources"); env.jersey().register(new PingResource()); env.jersey().register(new StudentResource(studentService, aerospikeClient, config)); env.healthChecks().register("DropwizardAerospikeHealthCheck", new DropwizardAerospikeHealthCheckResource(config)); } }
Project Setup:
For Java Setup, please refer to: Java Setup
For Maven Setup, please refer to: Maven Setup
For Git and Project Setup, please refer to: Git and Project Setup
For Maven Setup, please refer to: Maven Setup
For Git and Project Setup, please refer to: Git and Project Setup
Run Application:
1. To run application in your IDE use:
Program arguments: src/main/resources/dropwizardaerospike.yml
Program arguments: src/main/resources/dropwizardaerospike.yml
2. To run JAR from command prompt:
Build jar by using command:
Build jar by using command:
mvn clean installRun JAR by using command in Project folder location:
java -jar target\DropwizardAerospike-1.0.0.jar src\main\resources\dropwizardaerospike.yml
API calls and results:
1. GET API to test cache
http://localhost:4000/student
When this API is called, if you check the logs, you can see that in the first call, as data is not present in cache, it is retrieved from the Database (HashMap). Also the cache gets populated with the data in Call 1. So in Call 2 and Call 3 you can see that data is being retrieved from the cache.
When the API is executed, you can check the data populated in the set in Aerospike using aql (Aerospike Query Language). For this, login to the Aerospike host. For instructions on how to login to Aerospike host using Oracle VM VirtualBox Manager, please refer to my post on: Aerospike Setup
Enter the following command to start using aql on the host:
Other than these APIs, this application has the following APIs:
1. GET API for Application Health Check:
http://localhost:4000/admin/healthcheck
2. GET API to Ping and test if the application is up and running:
http://localhost:4000/ping
3. POST API to Ping and test if the application is up and running:
http://localhost:4000/ping
JSON Request Body:
You can click on individual links on this page to see different application metrics.
When this API is called, if you check the logs, you can see that in the first call, as data is not present in cache, it is retrieved from the Database (HashMap). Also the cache gets populated with the data in Call 1. So in Call 2 and Call 3 you can see that data is being retrieved from the cache.
When the API is executed, you can check the data populated in the set in Aerospike using aql (Aerospike Query Language). For this, login to the Aerospike host. For instructions on how to login to Aerospike host using Oracle VM VirtualBox Manager, please refer to my post on: Aerospike Setup
Enter the following command to start using aql on the host:
aql -T 10000Query the set student, that we created in the application above, using:
select * from test.studentOther useful aql queries:
show namespaces show sets show binsYou can also check the reads and writes in Aerospike using the Aerospike Management Console (AMC). For instructions on how to access AMC, please refer to my post on: Aerospike Setup
Other than these APIs, this application has the following APIs:
1. GET API for Application Health Check:
http://localhost:4000/admin/healthcheck
2. GET API to Ping and test if the application is up and running:
http://localhost:4000/ping
3. POST API to Ping and test if the application is up and running:
http://localhost:4000/ping
JSON Request Body:
{ "input": "ping" }4. GET Admin API to see application metrics: Dropwizard provides this Admin API. As I have set adminContextPath as: /admin in dropwizardmongodb.yml the link I need to use in any browser is: http://localhost:4000/admin/
You can click on individual links on this page to see different application metrics.
Conclusion and GitHub link:
In this post I have shown you how you can integrate
your existing Dropwizard Application with Aerospike and perform
basic operations on Aerospike. The code used in this post is
available on GitHub.
Learn the most popular and trending technologies like Machine Learning, Chatbots, Angular 5, Internet of Things (IoT), Akka HTTP, Play Framework, Dropwizard, Docker, Elastic Stack, Netflix Eureka, Netflix Zuul, Spring Cloud, Spring Boot, Flask and RESTful Web Service integration with MongoDB, Kafka, Redis, Aerospike, MySQL DB in simple steps by reading my most popular blog posts at Software Developer Central.
If you like my post, please feel free to share it using the share button just below this paragraph or next to the heading of the post. You can also tweet with #SoftwareDeveloperCentral on Twitter. To get a notification on my latest posts or to keep the conversation going, you can follow me on Twitter or Instagram. Please leave a note below if you have any questions or comments.
Learn the most popular and trending technologies like Machine Learning, Chatbots, Angular 5, Internet of Things (IoT), Akka HTTP, Play Framework, Dropwizard, Docker, Elastic Stack, Netflix Eureka, Netflix Zuul, Spring Cloud, Spring Boot, Flask and RESTful Web Service integration with MongoDB, Kafka, Redis, Aerospike, MySQL DB in simple steps by reading my most popular blog posts at Software Developer Central.
If you like my post, please feel free to share it using the share button just below this paragraph or next to the heading of the post. You can also tweet with #SoftwareDeveloperCentral on Twitter. To get a notification on my latest posts or to keep the conversation going, you can follow me on Twitter or Instagram. Please leave a note below if you have any questions or comments.
Comments
Post a Comment