Dropwizard Redis Integration

Dropwizard Redis Integration

Introduction

        I was searching online for an end to end Dropwizard Redis Integration tutorial and all I could find were a few GitHub projects with very minimal documentation. Dropwizard and Redis are being used extensively in many startups and other technology giants but there is not a single end to end tutorial online. Hence I decided to come up with this post. This post will give you a comprehensive overview of Dropwizard and Redis Integration. 

        Redis is an open-source in-memory database project implementing a distributed, in-memory key-value store with optional durability. Hence Redis is used for caching data so that it can be retrieved very quickly. So Redis is commonly used to store frequently used data from the Database so that it can be retrieved quickly and improve the application performance by not making frequent database calls. So in many cases we may need to store custom Java Objects in the Redis cache.
Most of you might have heard of Jedis – the Java Redis Client Library. However there is lot of custom code that needs to be written in Jedis in order to cache a custom Java Object. So I will not be using Jedis instead I will be using Redisson which is also a Java Redis Client. In fact Redisson is currently being used in many startups and big enterprises. You can read more about Redisson and the organizations using it at the Redisson Website.

The advantages of using Redisson are:
  1. It can serialize/deserialize any custom Java object.
  2. It can manage the Redis connection.
  3. It can work with Redis asynchronously.
  4. It supports many popular codecs like Jackson JSON, Smile, CBOR, Avro, MsgPack, Kryo, FST, LZ4, Snappy and JDK Serialization.
In this tutorial, we will be integrating a Dropwizard Application with Redis using Redisson.

        Dropwizard is one of the most popular and most used frameworks for building microservices.
I am assuming that you have a Basic Knowledge of Dropwizard and have a Basic Dropwizard Application running in your machine. If not, please check my blog post on Basic Dropwizard Application by going to the link: Dropwizard Tutorial.

Requirements to Run the Application:
  1. Java
  2. Maven
  3. Redis
  4. IDE of your choice
Redis should be setup and running in your machine. To setup, run and test if Redis is working fine, please refer to my post on: Redis Setup.

Once you have a Basic Dropwizard Application and Redis up and running in your machine, here are the additional steps required to add the Redis integration.

Step 1: Maven Dependencies to be added in pom.xml

Add Redisson to the project by using the following dependency in the pom.xml:
<dependency>
    <groupId>org.redisson</groupId>
    <artifactId>redisson</artifactId>
    <version>3.7.0</version>
</dependency>
com.fasterxml.jackson library dependencies are being added as part of the Redisson library that we added above. I am using org.apache.commons library dependencies for a few operations in this project. Both Dropwizard and Apache Commons have jackson-databind libraries as part of it. Hence to avoid conflict in the libraries, I have excluded jackson-databind that are part of Dropwizard and Apache Commons. I have explicitly added com.fasterxml.jackson library dependencies in the pom.xml to avoid any issues during application startup. Hence here is the full pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.aj.dropwizardredis</groupId>
 <artifactId>DropwizardRedis</artifactId>
 <version>1.0.0</version>


 <properties>
  <dropwizard.version>1.3.4</dropwizard.version>
        <redisson.version>3.7.0</redisson.version>
        <jackson.version>2.9.5</jackson.version>
        <jdk.version>1.8</jdk.version>
  <packaging>jar</packaging>
 </properties>

 <dependencies>

  <dependency>
            <groupId>io.dropwizard</groupId>
            <artifactId>dropwizard-core</artifactId>
            <version>${dropwizard.version}</version>
            <exclusions>
                <exclusion>  <!-- declare the exclusion here -->
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.redisson</groupId>
            <artifactId>redisson</artifactId>
            <version>${redisson.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-collections4</artifactId>
            <version>4.1</version>
            <exclusions>
                <exclusion>  <!-- declare the exclusion here -->
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>${jackson.version}</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>${jackson.version}</version>
        </dependency>

 </dependencies>

 <build>
  <plugins>
   <plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.0</version>
    <configuration>
     <source>1.8</source>
     <target>1.8</target>
    </configuration>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>2.4</version>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <configuration>
     <createDependencyReducedPom>true</createDependencyReducedPom>
     <filters>
      <filter>
       <artifact>*:*</artifact>
       <excludes>
        <exclude>META-INF/*.SF</exclude>
        <exclude>META-INF/*.DSA</exclude>
        <exclude>META-INF/*.RSA</exclude>
       </excludes>
      </filter>
     </filters>
    </configuration>
    <executions>
     <execution>
      <phase>package</phase>
      <goals>
       <goal>shade</goal>
      </goals>
      <configuration>
       <transformers>
        <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
         <mainClass>com.aj.dropwizardredis.DropwizardRedisApplication</mainClass>
        </transformer>
       </transformers>
      </configuration>
     </execution>
    </executions>
   </plugin>
  </plugins>
 </build>
</project>

Step 2: Redisson Managed Object

Most services involve objects which need to be started and stopped: thread pools, database connections, etc. Dropwizard provides the Managed interface for this. Managed interface is a powerful and very useful concept that you should be aware of in Dropwizard. You can either have the class in question implement the start() and stop() methods, or write a wrapper class which does so. Adding a Managed instance to your service’s Environment ties that object’s lifecycle to that of the service’s HTTP server. Before the server starts, the start() method is called. After the server has stopped (and after its graceful shutdown period) the stop() method is called. Redisson manages the Redis conncetion. Hence we will configure Redisson as a Managed object. The code required to do that is present in RedissonManaged.java. Here is the code:
package com.aj.dropwizardredis;

import io.dropwizard.lifecycle.Managed;
import org.redisson.api.RedissonClient;

public class RedissonManaged implements Managed {

    private RedissonClient redissonClient;

    public RedissonManaged(RedissonClient redissonClient) {
        this.redissonClient = redissonClient;
    }

    @Override
    public void start() throws Exception {
    }

    @Override
    public void stop() throws Exception {
        redissonClient.shutdown();
    }
}

Step 3: Student POJO

In this post I am using a basic Student POJO as an example. Student has 3 attributes: name, universityId and subjectSpecialization

Here is Student.java:
package com.aj.dropwizardredis.domain;

public class Student {

    private String name;
    private String universityId;
    private String subjectSpecialization;

    public Student(String name, String universityId, String subjectSpecialization) {
        this.name = name;
        this.universityId = universityId;
        this.subjectSpecialization = subjectSpecialization;
    }

    public Student() {
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getUniversityId() {
        return universityId;
    }

    public void setUniversityId(String universityId) {
        this.universityId = universityId;
    }

    public String getSubjectSpecialization() {
        return subjectSpecialization;
    }

    public void setSubjectSpecialization(String subjectSpecialization) {
        this.subjectSpecialization = subjectSpecialization;
    }

    @Override
    public String toString() {
        return "Student{" +
                "name='" + name + '\'' +
                ", universityId='" + universityId + '\'' +
                ", subjectSpecialization='" + subjectSpecialization + '\'' +
                '}';
    }
}

Step 4: CacheConfigManager Class

In this class, I am building a new Student Cache. In this class I am also defining a method which returns student data when a key is provided. Also observe here how I am using Redisson Distributed Collection RMapCache. I am also setting cache expiration time for each entry in the cache as 30 minutes. I will be giving more details about Redisson Distributed Collections in Step 9. Here is the class: CacheConfigManager.java
package com.aj.dropwizardredis.cache;

import com.aj.dropwizardredis.domain.Student;
import com.aj.dropwizardredis.service.StudentService;
import org.apache.commons.collections4.CollectionUtils;
import org.redisson.api.RMapCache;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.concurrent.TimeUnit;

public class CacheConfigManager {

    private static final Logger logger = LoggerFactory.getLogger(CacheConfigManager.class);

    private static CacheConfigManager cacheConfigManager = new CacheConfigManager();

    public static CacheConfigManager getInstance() {
        return cacheConfigManager;
    }

    //Logic For Student Cache
    public Student getStudentDataFromCache(String key, StudentService studentService,
                                           RMapCache<String, Student> studentRMapCache) {
        try {
            Student student;
            if(CollectionUtils.isEmpty(studentRMapCache.keySet())){
                student = studentService.getFromDatabase(key);
                //Cache will expire after 30 minutes
                studentRMapCache.put(key, student, 30, TimeUnit.MINUTES);
            }
            else{
                if(studentRMapCache.containsKey(key)){
                    student = studentRMapCache.get(key);
                }
                else{
                    student = studentService.getFromDatabase(key);
                    //Cache will expire after 30 minutes
                    studentRMapCache.put(key, student,30, TimeUnit.MINUTES);
                }
            }
            logger.info("All Entries in Student map: {}",studentRMapCache.readAllEntrySet());
            return student;
        } catch (Exception e) {
            logger.error("Error Retrieving Elements from the Student Cache"
                    + e.getMessage());
            return null;
        }
    }
}

Step 5: Student Service Class

        In this post, to keep things simple, I am not using a database instead I am creating and using a HashMap as my database. This HashMap has universityId as key and the Student Object as Value. In this class I have created a method getFromDatabase which will be called whenever the cache does not have the key that I am looking for.

Here is StudentService.java:

package com.aj.dropwizardredis.service;

import com.aj.dropwizardcache.domain.Student;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;
import java.util.Map;

public class StudentService {

    private static final Logger logger = LoggerFactory.getLogger(StudentService.class);

    public StudentService() {
    }

    public static Student getFromDatabase(String universityId) {
        Student student1 = new Student("Jim", "S100", "Science");
        Student student2 = new Student("Steve", "M101", "Maths");
        Student student3 = new Student("Mark", "P102", "Physics");
        Map<String, Student> database = new HashMap<>();
        database.put("S100", student1);
        database.put("M101", student2);
        database.put("P102", student3);
        logger.info("Database called for: {}", universityId);
        return database.get(universityId);
    }
}

Step 6: API to test Cache

I have created a GET API in Dropwizard which will call method to get student data from cache. I have written code to call this method 3 times. So during the invocation, if you check the application logs, it will clearly show that the data is retrieved from the database during the first call. During the first call the cache will also get populated. Hence the subsequent calls will retrieve data from the Cache.

Here is StudentResource.java:
package com.aj.dropwizardredis.resource;

import com.aj.dropwizardredis.cache.CacheConfigManager;
import com.aj.dropwizardredis.domain.Student;
import com.aj.dropwizardredis.service.StudentService;
import com.codahale.metrics.annotation.Timed;
import org.redisson.api.RMapCache;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.HashMap;
import java.util.Map;

@Path("/student")
@Produces(MediaType.APPLICATION_JSON)
public class StudentResource {

    private static final Logger logger = LoggerFactory.getLogger(StudentResource.class);

    private static StudentService studentService;
    private static RMapCache<String, Student> map;

    public StudentResource(StudentService studentService, RMapCache<String, Student> map) {
        this.studentService = studentService;
        this.map = map;
    }

    @Timed
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response cache() {
        logger.info("In StudentResource.cache()...Get Student Data");
        //On the first call, data will be fetched from DB and
        //cache will be populated with the corresponding student record
        //On all subsequent calls, data will be returned from the cache
        for (int i = 1; i < 4; i++) {
            getStudentData(i);
        }
        Map<String, String> response = new HashMap<>();
        response.put("message", "Student Data has been retrieved");
        return Response.ok(response).build();
    }

    private void getStudentData(int i) {
        logger.info("********** Call " + String.valueOf(i) + " Started **********");
        logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentDataFromCache("S100",studentService, map));
        logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentDataFromCache("M101",studentService, map));
        logger.info("Call " + String.valueOf(i) + ": {}", CacheConfigManager.getInstance().getStudentDataFromCache("P102",studentService, map));
        logger.info("********** Call " + String.valueOf(i) + " Ended **********");
    }
}

Step 7: Configuration in yml file and in the Configuration class

In the yml file I am setting the name of the Student Cache Key as: student. I am also setting the Application Name as DropwizardRedis, Application server port as 4000 and adminContextPath as: /admin
Here is the full yml file dropwizardredis.yml:
logging:
  level: INFO
  
  appenders:
    - type: console
      threshold: ALL
      timeZone: IST
      
server:
  type: simple
  applicationContextPath: /
  adminContextPath: /admin
  connector:
      port: 4000
      type: http

appName : DropwizardRedis

studentCacheKey: student
Getters and setters for the properties in the yml file are created in the Configuration class.
Here is the file: DropwizardRedisConfiguration.java:
package com.aj.dropwizardredis;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.dropwizard.Configuration;

@JsonIgnoreProperties(ignoreUnknown = true)
public class DropwizardRedisConfiguration extends Configuration {

    @JsonProperty
    public String appName;

    @JsonProperty
    public String studentCacheKey;

    public String getAppName() {
        return appName;
    }

    public void setAppName(String appName) {
        this.appName = appName;
    }

    public String getStudentCacheKey() {
        return studentCacheKey;
    }

    public void setStudentCacheKey(String studentCacheKey) {
        this.studentCacheKey = studentCacheKey;
    }
}

Step 8: Create Health Check Class 

In the Health Check Resource, I am performing a basic check of the Application Name retrieved from the file: dropwizardredis.yml
Here is the file: DropwizardRedisHealthCheckResource.java:
package com.aj.dropwizardredis.resource;

import com.aj.dropwizardredis.DropwizardRedisConfiguration;
import com.codahale.metrics.health.HealthCheck;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class DropwizardRedisHealthCheckResource extends HealthCheck {

    private static final Logger logger = LoggerFactory.getLogger(DropwizardRedisHealthCheckResource.class);

    private static String appName;

    public DropwizardRedisHealthCheckResource(DropwizardRedisConfiguration dropwizardRedisConfiguration){
       this.appName = dropwizardRedisConfiguration.getAppName();
    }

    @Override
    protected Result check() throws Exception {
        logger.info("App Name is: {}", appName);
        if("DropwizardRedis".equalsIgnoreCase(appName)) {
            return Result.healthy();
        }
        return Result.unhealthy("DropwizardRedis Service is down");
    }
}

Step 9: Create the Application Class 

Everything comes together in the application class. We register the resources, health check and Redisson Managed Object in the application class. 
There are a couple of important things to observe in the Application Class:
1. Creation of RedissonClient: I have created the RedissonClient in the most simple way by using the following code:
RedissonClient redissonClient = Redisson.create();
This will connect to Redis localhost at the default port 6379
Different configurations can be passed to the Redisson object’s create method. This could be configurations to have it connect to a different port, or to connect to a Redis cluster.
This configuration could be done programmatically in the Java code or loaded from an external configuration file which could be a JSON file or YAML file.
For detailed instructions about the usage of these configurations please check: Redisson Website.
2. Redisson Distributed Collections: There are multiple Redisson Distributed Collections that can be used to store data and thus acting as a cache. I am using RMapCache in this Application so that I have eviction support.
Map object with eviction support implements org.redisson.api.RMapCache which extends java.util.concurrent.ConcurrentMap interface.
Here is the code where I am creating RMapCache that I use as the Student Cache in this application. I am also capturing a key expire event in the second line of code and logging it:
RMapCache<String, Student> studentMap = redissonClient.getMapCache(config.getStudentCacheKey());
        studentMap.addListener((EntryExpiredListener<String, String>) entryEvent -> {
            logger.info("Expired key is: {}, Expired value is: {}", entryEvent.getKey(),entryEvent.getValue());
        });
There are multiple Redisson Distributed Collections that you can use as a cache in your application. For detailed instructions about the usage of these Redisson Distributed Collections please check: Redisson Distributed Collections.
Redisson also provides Distributed Objects which can be used in Redis Integration. For detailed instructions about the usage of these Redisson Collections please check: Redisson Distributed Objects.
Here is the file: DropwizardRedisApplication.java:
package com.aj.dropwizardredis;

import com.aj.dropwizardredis.domain.Student;
import com.aj.dropwizardredis.resource.DropwizardRedisHealthCheckResource;
import com.aj.dropwizardredis.resource.PingResource;
import com.aj.dropwizardredis.resource.StudentResource;
import com.aj.dropwizardredis.service.StudentService;
import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;
import org.redisson.Redisson;
import org.redisson.api.RMapCache;
import org.redisson.api.RedissonClient;
import org.redisson.api.map.event.EntryExpiredListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class DropwizardRedisApplication extends Application<DropwizardRedisConfiguration> {

    private static final Logger logger = LoggerFactory.getLogger(DropwizardRedisApplication.class);

 public static void main(String[] args) throws Exception {
  new DropwizardRedisApplication().run("server", args[0]);
 }

    @Override
    public void initialize(Bootstrap<DropwizardRedisConfiguration> b) {
    }

 @Override
 public void run(DropwizardRedisConfiguration config, Environment env)
   throws Exception {
        //Connect to 127.0.0.1:6379 by default
     RedissonClient redissonClient = Redisson.create();
     RedissonManaged redissonManaged = new RedissonManaged(redissonClient);
        env.lifecycle().manage(redissonManaged);
        RMapCache<String, Student> studentMap = redissonClient.getMapCache(config.getStudentCacheKey());
        studentMap.addListener((EntryExpiredListener<String, String>) entryEvent -> {
            logger.info("Expired key is: {}, Expired value is: {}", entryEvent.getKey(),entryEvent.getValue());
        });
     StudentService studentService = new StudentService();
     logger.info("Registering RESTful API resources");
  env.jersey().register(new PingResource());
        env.jersey().register(new StudentResource(studentService,studentMap));
  env.healthChecks().register("DropwizardCacheHealthCheck",
    new DropwizardRedisHealthCheckResource(config));
 }
}

Project Setup:

For Java Setup, please refer to:  Java Setup
For Maven Setup, please refer to: Maven Setup
For Git and Project Setup, please refer to: Git and Project Setup

Run Application:

1. To run application in your IDE use:
    Program arguments: src/main/resources/dropwizardredis.yml
2. To run JAR from command prompt:
     Build jar by using command:   
   mvn clean install
     Run JAR by using command in Project folder location:
   java -jar target\DropwizardRedis-1.0.0.jar src\main\resources\dropwizardredis.yml

API calls and results:

1. GET API to test cache
When this API is called, if you check the logs, you can see that in the first call, as data is not present in cache, it is retrieved from the Database (HashMap). Also the cache gets populated with the data in Call 1. So in Call 2 and Call 3 you can see that data is being retrieved from the cache.
When the API is executed, the RMapCache: student will be created in Redis. You can use Redis CLI and execute the following command to check the RMapCache: student created:
keys *
Execute the following command to check the keys and values in the RMapCache: student
HGETALL student
Also as I have logged the cache eviction event, when the cache expires you can see in the logs the keys and values that have been evicted.

Other than these APIs, this application has the following APIs:

1. GET API for Application Health Check:
http://localhost:4000/admin/healthcheck
2. GET API to Ping and test if the application is up and running:
    http://localhost:4000/ping
3. POST API to Ping and test if the application is up and running:
    http://localhost:4000/ping  
    JSON Request Body:
    {
     "input": "ping"
    }
4. GET Admin API to see application metrics: Dropwizard provides this Admin API. As I have set adminContextPath as: /admin in dropwizardmongodb.yml the link I need to use in any browser is: http://localhost:4000/admin/
You can click on individual links on this page to see different application metrics.

Conclusion and GitHub link:

    In this post I have shown you how you can integrate your existing Dropwizard Application with Redis using Redisson and perform basic operations on Redis. The code used in this post is available on GitHub.
    Learn the most popular and trending technologies like Machine Learning, Chatbots, Angular 5, Internet of Things (IoT), Akka HTTP, Play Framework, Dropwizard, Docker, Elastic Stack, Netflix Eureka, Netflix Zuul, Spring Cloud, Spring Boot, Flask and RESTful Web Service integration with MongoDB, Kafka, Redis, Aerospike, MySQL DB in simple steps by reading my most popular blog posts at Software Developer Central.
    If you like my post, please feel free to share it using the share button just below this paragraph or next to the heading of the post. You can also tweet with #SoftwareDeveloperCentral on Twitter. To get a notification on my latest posts or to keep the conversation going, you can follow me on Twitter. Please leave a note below if you have any questions or comments.

Comments

  1. For those of us who get stuck in legacy code world occasionally, this was invaluable to me, thanks for taking the time to write it up.

    ReplyDelete

Post a Comment

Popular Posts

Golang gRPC Microservice

Dropwizard MySQL Integration Tutorial

Asynchronous Processing (@Async) in Spring Boot