July 31, 2017

Traditional managed file transfer (MFT) solutions are good at one thing: performing the core file exchange and transmitting file data. Since MFT came of age in the mid-2000s, the landscape has rapidly changed toward cloud-hosted applications and IT infrastructures have become increasingly diverse and distributed. What that means is that functional requirements are also evolving. There is more than the simple need to transfer unaltered files from the source to destination systems. The file transfer must perform in-transit processing on the file data itself as records or events distributed across your diverse and specialized SaaS and enterprise applications.

Microservices based API integration

RoboMQ MFT service allows you to transfer files between file systems using a variety of standard file transfer protocols including FTP, SFTP, SCP, et al, over various file systems, S3, Object Stores, Cloud Storage, and more. Merging both the core file handling and processing with API integration capabilities opens a door to unlimited possibilities.

Within the Control Layer, all file handling and processing tasks are performed by the RoboMQ functional components, i.e. microservices which are depicted as hexagons in the figure above. These components can be assembled within a messaging fabric that together creates an event-driven MFT workflow that is robust, scalable, and reliable. Any number of tasks can be added within the workflow. For instance, file transformation, validation, encryption, or any custom task of choice can be easily added. In addition, given the protocol-agnostic capabilities of the RoboMQ platform, file processing can expand away from conventional SFTP or SCP file transfer protocols to integrate with REST, SOAP, JSON or XML APIs. Below are just some of the countless integration use cases to inspire you.

File to event processing:

RoboMQ microservices offer a variety of ways to add event-driven functionality into your workflow. Take a CSV file for example, a simple file format used to store tabular data. From that source file, we offer microservices with the ability to break each individual row of the file into records. Those records, in turn, are processed as events from that point forward throughout the workflow.

Event aggregation to file:

HR or payroll processing solutions require retrievable employee identity and job-related attributes. As many finance and human resources leaders including ADP and Workday^® offer the latest REST API integration technologies, workflows could be customized to query the resources and seamlessly aggregate into output files for transfer to Point Of Sale (POS) or HR management system.

File to stream processing:

Another increasingly popular use case is stream processing. Given the pluggable broker support with RoboMQ, using Apache Kafka as a messaging transport is a great fit for stream processing apps. Events or data can be captured as a stream from a source file, then fed to other data systems such as relational databases, object stores or data warehouses. With HTTP streaming now a reality adding an HTTP connector into the mix can create a media content producer from any source audio/video files. Or in some cases, file transfers over HTTP may simply be useful for uploading large files to a destination server.

Additional microservices can be joined into existing workflows, to flag and alert on any invalid or missing content through email, SMS, or phone call. This can save managers and system administrators countless hours by no longer having to track down the origin of errors.

In summary, RoboMQ is a middleware with no protocol of its own but can support any protocol, including cross conversion among them. RoboMQ provides seamless integration of file-based systems, SaaS and cloud systems, and diverse applications all done over a distributed system. Finally, being an enterprise-grade MFT platform, all the management, error handling and recovery features are built in to provide a reliable and secure file transfer.

For more details on RoboMQ Managed File Transfer, please checkout out our Managed File Transfer page.