Integration of Data Flow Application for a Multi-Vendor Automotive Marketplace on Magento

Challenges

Once WheelerShip started adding new vendors to their Magento marketplace, the company faced new challenges connected to data processing on its website. Adding more components would make the existing infrastructure more complex. Plus, the product information from vendors needed to be fetched, processed and converted into a unified format for the correct display on the website.

The major problem was that the WheelerShip website was initially developed as a smaller online store, and processes within the website were supposed to work differently than a bigger marketplace required. So the client asked us to streamline the website’s existing workflow and harmonize vendors’ data processing without recreating the whole website infrastructure from scratch. In the long run, this should also facilitate adding more vendors to the marketplace and connecting other systems.

Discovery Stage

Fishbowl ERP x Magento x Custom App.We started the project by exploring the existing setup and discovering how the client received the data from vendors.

Different teams had already been supporting systems within the WheelerShip infrastructure before the client reached out to us. These systems were exchanging data directly with one another:

• • 1️⃣ Magento, the platform on which the website was developed.
  • 2️⃣ Fishbowl ERP*, where the client managed products and product descriptions.
  • 3️⃣ A custom application responsible for reporting, custom logic, and competitors’ price monitoring.

  All three systems communicated like that:

Magento Multi-Vendor Marketplace

Also, WheelerShip received vendors’ product data in different formats and files, for example, an Excel sheet or text document with a different number of attributes. So, connecting each vendor directly to Magento would require a high level of customization, which is time-consuming and expensive.

But even after the customization, the system would be hard to maintain. Plus, adding more vendors in the future would require the same amount of effort, time, and costs. With too many vendors the system would also be overloaded, resulting in poor performance.

To summarize, the system WheelerShip had at the time could not support several vendors and properly extract, synchronize and send different data formats to Magento and Fishbowl ERP. Our task was to make all of the existing systems communicate smoothly between each other and with new vendors added to Magento with minimum customization.

Solutions

Development of Data Flow Application for Magento Store

Analyzing existing processes within WheelerShip infrastructure allowed us to arrive at the decision that an intermediary file processing application connecting vendors’ data to Magento and Fishbowl ERP is the best solution. This file processing app could extract the data and convert it to the formats required for these two systems.

We offered them our internal solution, which was already developed for similar requirements. This solution is called Data Flow Application (DFA) based on the MACH architecture*. And it is a part of our R&D activities. DFA serves as middleware between external systems, in this case, Fishbowl ERP and Magento, and vendors.

Custom ERP and eCom adapter connect Magento and Fishbowl to DFA via APIs**. Adapters identify data transfer formats and convert heterogeneous data into a unified language. Then adapters send the information to the core of the DFA, called the brain of the application, which routes requests among different systems. The brain of the application knows what kind of data and where to send it from one system to another, and how often.

DFA lets vendors send their data to Magento also via custom adapter. We needed to deal with different data formats each time we added a new vendor to the system, so we developed adapters for each particular vendor.

We developed custom adapters that could take data from Excel sheets or APIs, transform it into a specific format, and send it to the ‘brain’. After the ‘brain’ analyzes the data and adjusts mapping attributes to the rules of Magento, the data finally goes to Magento.

The following diagram shows how DFA works:

Vendor Portal Development

Currently, we are also working on creating a vendor portal where vendors can specify the way they communicate with WheelerShip, make the necessary configurations and settings (such as price margin, shipping options, cut-off time etc), track the progress of the information exchange, and solve any issues with products without involving WheelerShip.