How to support your microservices architecture with iPaaS

In this article, consider how an integration platform as a service (iPaaS) can support various architectural approaches, including microservices. Learn how Dell Boomi allows you to modularize your integration services.

Read this article to know what exactly is microservices and how does it apply to the world of integration? Will it really provide me with benefits or will it be more trouble than it’s worth?

Content Summary

WHAT IS MICROSERVICES?
DOES AN INTEGRATION LAYER NEED TO FOLLOW THE MICROSERVICES APPROACH?
INDEPENDENTLY DESIGNING AND DEPLOYING SERVICES
REVISITING THE CHARACTERISTICS OF MICROSERVICES
DECENTRALIZED GOVERNANCE
INFRASTRUCTURE AUTOMATION
EXAMPLE SCENARIO
BOOMI DOCKER ATOM
BOOMI ENVIRONMENT
KUBERNETES FOR DOCKER ATOMS
HOW BOOMI CAN HELP

Microservices – a word often used in IT these days.

You may wonder what exactly microservices is and how it applies to the world of integration. Will it really provide me with benefits or will it be more trouble than it’s worth?

Microservices is a broad technology topic. This brief does not aim to state whether a microservices approach is good or bad. Sometimes a microservices approach makes sense for your situation, but in other cases, it simply isn’t the right technology for a given use case.

In this brief, we focus only on how microservices relates to modern, cloud-based integration. My aim is to show you that an integration platform as a service (iPaaS) like Boomi can support many different architectural approaches, including microservices. Let’s get started.

WHAT IS MICROSERVICES?

Microservices is not a product or pre-baked solution. It is an approach or framework designed to make life easier for IT people. In short, the microservices architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and using different data storage technologies.

This architectural style has many characteristics, some of which would make sense to apply to the integration layer:

• Componentization via services
• Organized around business capabilities
• Products, not projects
• Design for failure
• Decentralized governance
• Infrastructure automation

The powerful yet complex thing about microservices is that it aims to give guidance on an approach that spans the complete software development life cycle. In fact, one could argue that it’s more about designing than building.

DOES AN INTEGRATION LAYER NEED TO FOLLOW THE MICROSERVICES APPROACH?

I have had customers ask, “Why would an integration layer need to follow the microservices approach? Doesn’t microservices apply only to business applications?”

This is partly true because integration, while the Boomi team is very proud to have engineered the marketleading integration platform, is by no means as business functionality-rich as your applications. An application is extremely rich in business rules and functions, which effectively run your business.

Hence, if you have a large application, then microservices might provide benefits, since you are separating business rules and functionality into separate life cycle-managed services. This keeps your application modular, reduces dependencies, and allows for different parts of your application to be developed at a different pace. These are many good things you would want when developing a core business application.

While critical, integration has far fewer unique business functions than applications. Data is fetched from the source, transformed, augmented, mapped to a new format, and sent to the target. There may be some protocol switching along the way to complicate it, but nothing is completely unique to your organization.

That is why rather than reinventing the wheel, organizations buy an integration platform. So why would you want to split this out into modular services?

Well, in some cases, this is a valid point. Even if your business applications are structured in a microservices approach, they can still talk to the one integration platform for data exchange. If the integration platform needs more capacity, simply scale it up or scale it out.

This approach has worked for many organizations and, in particular, Boomi customers, since Boomi already allows you to design and deploy integration processes and APIs completely independent of each other. Let’s go through an example.

INDEPENDENTLY DESIGNING AND DEPLOYING SERVICES

Let’s say you have three integration processes – A, B, and C. Each process is segregated according to business function, developed and deployed independently. Each process has an “API” definition component sitting in front of it so that it can be called through an API (e.g., REST).

The combination of the API and process is what makes it a “service.” Both the API and the process (the process may be more than one and there may be subprocesses as well) are grouped into a “Package,” a deployable unit of work. Package A is managed by Team A, Package B is managed by Team B, and so forth. Process A is an orchestration process so it has to call Process B and Process C to satisfy the incoming request from the consumer (see Figure 1).

Now, let’s say that your organization has decided to replace Business System Y with a cloud-based system. This change has also required a slight modification to Process C. But the change can be done independently of Process A or Process B. API C has not changed, so none of the links are broken. Additionally, this change can be done with no need for any downtime of the runtime, hence no disruption to any of the other services (see Figure 2).

Another approach for achieving this decoupling is through using what’s known as Atom Queues. An Atom Queue is a lightweight asynchronous messaging feature within Boomi that allows decoupled inter-process communication.

If we applied Atom Queues to the same example, Process A would put a message onto Queue B, which is subscribed to by Process B. Process A would also put a message onto Queue C, which is subscribed to by Process C. If Process C needs to be updated, Process A does not have to change its logic in any way. It continues to put messages on Queue C and the new version of Process C will process it accordingly (see Figure 3).

In fact, Atom Queues are a great way not only to achieve this interprocess decoupling but to design processes for being resilient to failure. If a process fails, put an event on an Atom Queue and return an HTTP response code, Message Code, and Message description. The queue has a set of listening processes ready to process this failure and take necessary remediation (see Figure 4).

If you already have a standard messaging platform within your organization, Boomi provides connectors to integrate into these systems (see Figure 5).

So in a nutshell, Boomi inherently allows you to modularize your integration services, hot-deploy them into existing runtimes, and manage the life cycle of these processes independent of one another.

You can also use the “forked execution” feature to run each service on a separate Java virtual machine, protecting them from failures. This is the sort of stuff you pay for when you buy an iPaaS – might as well use what you pay for!

REVISITING THE CHARACTERISTICS OF MICROSERVICES

So let’s revisit the list of characteristics that define microservices and see how many of them we’ve covered:

1. Componentization via services: This is done through the combination of designing decoupled processes and service-enabling them through API components.
2. Organized around business capabilities: This is done via the good design of segregating business capabilities into separate packages.
3. Products, not projects: Each package can be a product, not a project, with its own unit tests and life cycle.
4. Design for failure: Use Atom Queues to decouple error handling.
5. Decentralized governance: Yet to be addressed.
6. Infrastructure automation: Yet to be addressed.

So if we go back to the question we started with (“Does an integration platform need to follow the microservices approach?”), the answer is that when it comes to Boomi, it already has many of the characteristics you need to implement integrations in a microservices approach. It’s up to you to design it in a way that leverages all of these characteristics.

Now, what about the last two characteristics? Decentralized governance and infrastructure automation? Let’s cover that next

DECENTRALIZED GOVERNANCE

This is probably the most debatable issue in microservices and integration. Purists claim that products such as an enterprise service bus or an iPaaS force centralized governance, hence they do not meet the decentralizedgovernance characteristic of microservices.

But that depends on how we define decentralized governance. Boomi’s development environment is a graphical drag-and-drop, low-code environment. While Boomi supports code development with Java or variants of Java such as Groovy, Boomi developers do not have an unlimited choice of programming languages. Also, services developed in Boomi can run only on Boomi runtimes (i.e., Atoms & Molecules) and they are accessed only through the Boomi AtomSphere design and administration platform.

So if you refer to decentralized governance as strictly having the ability to use any technologies or programming languages of your choice to build services, then Boomi does not fit this criterion.

Does that mean Boomi is straight up not a decentralized solution? Well, not quite. Boomi allows decentralization of services from an ownership and infrastructure perspective. But you do have to use Boomi’s centralized design and management platform. And there is good reason for this. Let me explain.

One of the things I alluded to earlier but haven’t explicitly called out is that one of the tenets of a microservices approach is “smart endpoints and dumb pipes.” This means that endpoint applications should be smart enough to make decisions on consuming and producing messages. The “pipe” should be just a “dumb” pipe that transports these messages from one smart application to another.

The question then is: How many smart applications do you have? Actually, the better question may be of your critical business applications: How many have you had the luxury (time, money, resources) of developing it to be smart enough for you? Additionally, how many cloudbased applications do you have which you have no control over how messages are consumed and produced?

The reality is, whether by design or misfortune, your business is likely running many systems that simply are not smart enough to handle data sharing and integrations. A company that has relied heavily on databases and stored procedures will know that it cannot just plug a “dumb” messaging layer to exchange data. And a company that has shifted many applications to cloudbased products will know that every cloud vendor built its software platforms differently, hence you need a “smarter” integration layerin the middle to do the translations for you.

Now if you take the essence ofthe phrase “smart endpoints and dumb pipes,” it says that your focus should be at the endpoint layer, not on the pipes layer. This is a good posture for an organization, since, as I mentioned earlier, integration is like plumbing. Data is fetched from the source, transformed, augmented, mapped to a new format, and sent to the target.

So then the question is:Would you want to manage two or three different technology sets to develop this layer? Would you want to access three or four different interfaces to maintain this layer?

Many organizations have realized that this approach is both expensive and inefficient. They prefer to use a centralized “single pane of glass” approach that standardizes integration service development, so you have more time to develop your endpoints. Hence, iPaaS is one of the fastest-growing cloud technologies in the entire market.

INFRASTRUCTURE AUTOMATION

I’ve spoken a lot about how Boomi can componentize and decouple services from a design perspective, but you’re probably burning to ask what about the infrastructure required to run these services? Does Boomi support runtimes in an automated, segregated fashion to support the microservices approach? If I want to execute different integration services running in different runtimes and scale them out independently, can I achieve this? The short answer is “yes.”

If you are familiar with Boomi, you’ll know that our runtime is called an “Atom.” A cluster of Atoms is called a “Molecule” (take you back to your chemistry days?).

If you’ve been following my narrative, you would have picked up that a lot of our customers already use the principles of microservices in their design but still run them on monolithic runtimes.

Different organizations have different reasons for this, but they mainly fall into one or more of the following categories:

• Our business is not complex enough to benefit from the microservices approach.
• Integration should be made easy — and it’s easier to maintain and deploy a single or clustered set of runtimes.
• Our organization does not have the maturity to manage a microservices infrastructure.
• Boomi’s default licensing favors Molecules over independent Atoms.

The first three reasons have a lot to do with the nature of an organization’s business and its assessment of the benefits that a microservices approach can bring (taking into consideration their current maturity).

The last reason is something the Boomi team can address for you if you are on the right variant of our Enterprise license.

Now, you may absolutely have a complex enough business that warrants a microservices approach. Your organization may already have the resources and skills to deploy a microservices architecture. For example, your organization may have one or more of the following:

• Geographic deployment requirements due to your business being global
• Dynamic scaling requirements that are seasonal (e.g., end-of-year sales or beginning-of-year enrollments)
• Shadow IT practices that require departmental autonomy

So how do you achieve this? Well, like all things in life, it comes down to choices, and in Boomi you have Atoms, Molecules, and Containerization to support these needs.

Let me give you an example.

EXAMPLE SCENARIO

Imagine you work for an Australian university. The university has decided to run its finance department separately in a shadow IT arrangement. And the department has decided to run everything in Microsoft Azure.

The university has also started a new educational program in Hong Kong to lure more overseas students. But it has Hong Kong privacy laws to comply with, hence it has decided to run all of its operations in a local instance of AWS in Hong Kong.

The main student and learning systems the university has maintained for many years still run on-premises and will stay there for the foreseeable future. So the university has both a hybrid architecture as well as geographic deployments, and your job is to architect an integration solution for it.

To further complicate the picture, you have just developed a new mobile application that allows students to access their timetables. The app allows students to modify and save their class schedules, so they can plan their weekly work and activities.

The challenge for you and the IT team is that the app is heavily used in the two weeks before and after the start of a semester. After that, the app is used only sparingly by students, so there needs to be dynamic scaling of the infrastructure.

As you can see from this example, each department or business unit has different needs, and the requirements for integration vary from department to department.

So what you really want in the true essence of an integration microservices architecture is independent, localized runtimes that support required behaviors needed by specific business tasks but with centralized management.

Now, as I alluded to earlier, the great thing about Boomi is that you don’t have to have a one-size-fits-all approach for your entire architecture. For some scenarios, you can simply use a single lightweight Atom as your integration services runtime. For other scenarios, you may want to configure a Molecule for high availability and scalability without the overhead of managing containers. And for other scenarios, you may want to use containerization technology for automation and dynamic scaling.

Here is a diagram of what your solution could look like (see Figure 6).

This diagram shows Boomi runtimes deployed across multiple computing environments in multiple configurations. As you can see, each department, region, or business domain has its own set of Atom runtimes with microservices running on them.

The finance department has decided to run just the single Atom but with virtual machine backups in case it needs to restore the runtime engine due to failure. The new business in Hong Kong has decided to use a highly available Molecule so business continuity is assured. The challenge is in student and learning systems, whereby they need to dynamically scale at certain periods during the year. This is an ideal situation, where Boomi runtimes (i.e., Atoms) are best used by leveraging container solutions.

Containers or containerization consists of an entire runtime environment: an application, plus all its dependencies, libraries and other binaries, and the configuration files needed to run the environment, bundled into one package.

The benefit of using containers is that the complexities of different computing environments and operating systems are abstracted away. In Boomi we provide this through Docker Atoms.

BOOMI DOCKER ATOM

Boomi provides a Docker image that allows you to deploy an Atom into a Docker container. Docker containers were initially born to support packaging applications.

Applications typically have dependencies such as config files, libraries, and sometimes even other software components such as databases or in-memory data grids.

The idea is that by packaging them into one container, you’ve created an independently deployable application unit, ready to be dropped into any environment that runs Docker.

Well, the same thing can be done with Atoms. The Atom luckily has no dependency on other software components, so the Docker image is the Atom executables along with all the config files and libraries, wrapped into a single deployable unit.

Not only do Docker Atoms allow for independent deployment of Atoms, but they ultimately allow for APIs and processes developed in Boomi to be independently run on these Atoms. For example, Figure 1 can now be modified to look like the following when using Docker Atoms (see Figure 7).

How granular you run your services is up to you. In this diagram, we have one API/Process combination running in one Docker Atom. But you will most likely have a collection of processes and APIs running in the Docker Atom to create your composite “product,” organized around a business capability.

Now, there’s one thing I need to make absolutely clear: It’s not Docker that allows the APIs and processes to be deployed independently. It’s a feature within Boomi called “Environment” that allows this.

BOOMI ENVIRONMENT

An “Environment” in Boomi is a feature that allows runtimes to be logically separated within your Boomi tenant. An environment can have one or more Atom, Molecule, or Docker Atom associated with it. Every time you deploy a package in Boomi, you pick an Environment to deploy to.

You may be getting a little confused because typically “environment” refers to a stage within a development life cycle, e.g., Dev, Test, or Prod. But in Boomi, while you use environments to segregate the stages within a development life cycle, you can also use them to logically separate microservices. For example, in the university scenario, you may have three environments for Finance (e.g., Finance Dev, Finance Test, Finance Prod), three environments for Hong Kong (e.g., Hong Kong Dev, Hong Kong Test, Hong Kong Prod), a number of environments for Student Management, and a number of environments for Timetable Management (see Figure 8).

You deploy your services into separate environments, and each environment will have one or more runtimes running to process execution of these services.

I explicitly call this out because some people may be under the misconception that you must have containerized Atoms to run microservices. That is not the case. You can run microservices on Atoms and Molecules.

The only difference is that if you run your microservices on an Atom or Molecule, all other microservices will run in the same shared runtime. You may ask, “If I build services based on microservices principles but run them in a shared runtime, is that microservices?” It’s debatable, but again it depends on your motivations for wanting a microservices approach. The good news is that Boomi has you covered for both sides of the argument.

If you have business requirements that need your microservices to run in dedicated runtimes, that’s when you can use Docker Atoms.

The catch is that with this approach you will start to create lots of environments with lots of Docker Atoms. Management overhead? Certainly. Can you reduce the overhead? Absolutely. This is where automation kicks in via Kubernetes.

KUBERNETES FOR DOCKER ATOMS

Kubernetes is an open-source, container-orchestration system for automating application deployment, scaling, and management. Simply put, it manages the life cycle of containers in an automated fashion. When you have only a few containers, you may not need this automation. But when you have tens or hundreds of containers, you’ll need an automation solution to create, configure, and destroy containers.

This is where Kubernetes really starts to play a crucial role. Kubernetes can monitor the load put on a cluster of containers. As the load increases, Kubernetes can make intelligent decisions on when to scale out the cluster. In the case of Boomi, Kubernetes would be spinning up a new Docker Atom, registering itself to the AtomSphere account, and assigning itself to a Boomi Environment.

This all happens dynamically, with zero user intervention. This is the power of Boomi when used in a containerized environment.

Furthermore, you want the containers to automatically register to a Boomi Environment so they start to communicate with the Boomi Platform. Kubernetes provides exactly that.

So let’s go back to one of the Australian university’s key requirements, which was to support the high volumes of load at the beginning of the semester for timetable access on the mobile app. What you want is for the Atoms dedicated to the Timetable Management System to dynamically scale out to support the high volumes of load (see Figure 9).

I’ve just walked you through a lot of information, so let’s summarize.

The university had a shadow IT structure in its finance department that required integrations to be run independently in Azure. But performance wasn’t critical; many of the integrations are scheduled and batch driven. Hence, a single Atom was deployed in Azure and periodically backed up.

Similarly, the education program in Hong Kong required in-country processing of data but also high availability of the runtimes, so the university ran a Molecule on AWS locally.

Finally, the student and learning management systems had integrations that required independently scalable runtimes, so the universisty used Docker with Kubernetes to create a dynamically scalable solution.

All services are still created on the Boomi AtomSphere Platform, and all are designed within the principles of microservices. The only difference is that some microservices are deployed to a shared Atom or Molecule and some are deployed to dedicated runtimes powered by Docker and Kubernetes.

Let’s revisit the characteristics of microservices one last time:

1. Componentization via services: This is done through the combination of building decoupled processes and service-enabling them through API components.
2. Organized around business capabilities: This is done via the good design of segregating business capabilities into separate packages.
3. Products, not projects: Each package can be a product, not a project, with its own unit tests and life cycle.
4. Design for failure: Use Atom Queues to decouple error handling.
5. Decentralized governance: This is debatable and depends on how you define “decentralized.” Boomi provides centralized management, for good reason.
6. Infrastructure automation: This is provided via a combination Boomi’s Docker Atoms and Kubernetes.

Architectural approaches like microservices are never just black and white. If you understand the shades of gray, then you’ll be able to strike a good balance between what is possible and what is required.

What’s important from your perspective is that there is a reason for adding another layer of complexity, and that should be to reduce overall net complexity.

It’s not just about technology, though. As mentioned earlier, there needs to be buy-in from your CIO and a funding model that plays nicely with the approach. This will require you to communicate, listen, and learn from all stakeholders — the good old-fashioned way!

HOW BOOMI CAN HELP

At Boomi, we have professional services consultants, enterprise architects, and integration specialists ready to help companies address any integration challenge big or small, including how to align your microservices strategy with your integration strategy.