Select Page

ERP and eCommerce –
Pros and Cons of Data Integration Techniques

Download Now

Introduction

Integration is not a new concept and has been around for ages. With the numerous programs and platforms that are necessary to run a successful business and the interdependencies between them, businesses for long have experimented with techniques to make systems talk to each other. Come 2016, integration still holds the key, only the end systems have evolved and so have the integration techniques.

In the context of e-commerce and ERP, integration plays a crucial role as it not only helps businesses run efficiently but also helps them drive omni-channel experience to their customers. In this eBook, we will explore the possible ways in which businesses can integrate their e-commerce and ERP systems and the pros and cons associated with each one of them.

Broadly, the data exchange between systems can happen in the following ways

  • Manual Data Transfer
  • Automated Data Transfer
  • Point-to-Point
  • Extract Transform Load (ETL)
  • Electronic Data Interchange (EDI)
  • API based Integrations
  • Hub-and-Spoke

For this eBook, we will limit ourselves to the automated data transfer techniques.

Extract Transform Load (ETL)

ETL was the first and most basic technique first used to automate and move away from the manual data transfer.

  • Extract – reads data from indicated source and pulls out the relevant subset of data (post filtering)
  • Transform – works with this acquired data – using rules or lookup tables, or creating combinations with other data – to convert it to the desired state. Surrogate keys or new key values are applied to similar data from different source systems prevent key collisions in the future and provide a cross reference across systems.
  • Load – writes the resulting data (either all of the subset or just the changes) to a target database, which may exist or is created.

While the ETL process can be kick-started both manually and automatically, it is usually automated using schedulers.

Pros and Cons

Pros Cons
  • Does not require a lot of investment (integration, infrastructure or otherwise)
  • Requires very little time to get up and running
  • Because the data is directly extracted and inserted into the database, data integrity is a concern
  • Because the data is directly extracted and inserted into the database, the application workflows are compromised
  • In case of issues, it is very difficult to debug, identify the cause and fix it
  • Does not guarantee real time updates – usually in situations when ETL is used the process is scheduled to run once a day during the off-peak hours
  • The solution cannot be customized to meet business specific requirements
  • Not a very secure way to transfer data between two systems

Commonly Used In

ETL is ideal in situations when large volumes of data are to be transferred between two systems and there are no workflows associated with the data that is transferred – like transferring historical data between systems.

Electronic Data Interchange (EDI)

EDI came next and revolutionized how transactional data was exchanged between two systems, or two parties (business and suppliers). EDI is a communication technology used to transmit data from one system to another.

  • EDI, the required data in the source system is translated into a standard EDI format using appropriate segments and data elements depending on how the internal data is mapped.
  • EDI uses private data network communications network called value-added networks (VANs) to transmit standardize transaction data between two systems.
  • After translating the data into the standard format, the connection is established with the destination system and the data is transmitted.
  • The information is then extracted from the transferred data and entered into the destination system.

In EDI the communication with the end applications can happen both at the database and application API level.

Pros Cons
  • This a secure way of exchanging information between the two systems
  • Because of the standard format of data exchange, it is very easy to replace one system with another
  • Because of the standard format of data exchange, this requires very little maintenance
  • Because of the standard format it is very easy to extend this by adding multiple senders or receivers (send data to multiple

    suppliers, etc.)
  • Implementation of the EDI infrastructure is expensive (both time and money)
  • If data is requested and inserted directly into the database
    • Because the data is directly extracted and inserted into the database, data integrity is a concern
    • Because the data is directly extracted and inserted into the database, the application workflows are compromised
  • Data synchronization in EDI is slow and time-consuming – the data is stored and then forwarded every scheduled interval (not very frequently). Because of this EDI cannot provide you real time or near real time experience.
  • Because of the non-real time nature of the communication, there is no scope for taking proactive decisions, identifying risks, or bypassing issues before they lead to unnecessary costs.
  • In case of issues, it is very difficult to debug, identify the cause and fix it · The solution cannot be customized to meet business specific requirements.

Commonly Used In

EDI is usually used in scenarios where transactional data has to be exchanged between various stakeholders on a non-real time basis. For example,

  • purchase order and invoice information flow between a distributor and trading partner,
  • order information flow between retailer and drop ship supplier, and more.

API based Integrations

APIs are EDIs of the 21st Century. API is a messaging format that allows data to be transmitted from one system to another. With APIs, the code is simplified and structured to clearly define how a program will interact with the rest of the applications. In the API based integrations, the integration application uses the APIs exposed by both the applications to extract data from and insert data into both the systems.

Pros Cons
  • Because APIs are used for communication,the database integrity is maintained.
  • Because APIs are used for communication, the data workflows are not compromised.
  • It is very easy to add checks to debug in case of errors.
  • Using APIs, the information can be exchanged between the two systems in real time or near real time.
  • The solution can be customized to meet business specific requirements
  • The information exchange can be encrypted to ensure data security.
  • Depending on the level of standardization, one system can be replaced with another system (say one ERP with another ERP)

    with little incremental effort.

  • The integration application once deployed requires very little maintenance.
  • API based communication make way for further automation
  • Ideal only in situations when you have to integrate two systems. As the number of systems increase so does the complexity of the setup.
  • API based integrations are time and resource intensive.
    • This is not true if the information to be exchanged between the two systems is fixed. In such cases, businesses develop the application once and spread the costs across multiple clients.

Commonly Used In

API based communication is probably the most widely used communication technique today. Most applications expose APIs to allow information exchange with other applications. This is usually used in situations where ongoing communication or data exchange is required between two systems. In addition, the information exchange does not always happen in both directions.

For example, even platforms like Facebook expose APIs to allow other applications to consume data residing with Facebook to enable them to leverage the data in other exciting ways.

Hub-and-Spoke

Hub-and-Spoke is not very different from API based integration – In API based integration (discussed above) we only had two systems whereas hub-and-spoke method extends the same concept but for more than two systems. In other words, in the hub-and-spoke integration method, there are multiple applications and a central integration application, usually hosted on the cloud, which manages the communication between these applications.

While the same model can be used to drive communication between just two systems, there are some limitations that sneak in because of the architecture (as it was designed for more than 2 systems).

Pros Cons
  • You can effectively manage information exchange between multiple applications from a centralized location
  • Very easy to add or remove applications
  • Because APIs are used for communication, the database integrity is maintained.
  • Because APIs are used for communication, the data workflows are not compromised.
  • It is very easy to add checks to debug in case of errors.
  • Using APIs, the information can be exchanged between the two systems in real time or near real time.
  • The information exchange can be encrypted to ensure data security.
  • The integration application once deployed requires very little maintenance.
  • Hub-and-Spoke solutions are usually cost effective
  • The solution cannot be customized to meet business specific requirements. Because the data exchanged is fixed, hub-and-spoke applications border on a data-mapping tool.
  • Since the data is copied to a third party cloud system, data security is a potential risk.
  • Since the data is exchanged via a third party cloud system, this adds another dependency to run your business.

Commonly Used In

Hub-and-Spoke method is ideal in situations where you have to simultaneously exchange information between multiple systems.

Conclusion

Data exchange techniques have evolved over the years and each method has its own application. The idea is not to recommend one method over other but to make our readers aware of each one of these methods and enable them to make an informed decision. Here is a quick overview of what we have discussed in the eBook.

Category ETL EDI Point-to-Point Hub-and-Spoke
Investment (Time and Money) Low High High Low
Turnaround Time Low High High Low
Data Integrity a Concer Y May Be N N
Application Workflows are Compromised Y May Be N N
Easy to Debug in case of Issue N N Y Y
Real time Updates N N Y Y
Supports Customization N N Y N
Security a Concern Y N N Y
Replace One System with Another N Y May Be Y
Scalability (easy to add more Systems) N Y N Y
Decision Making (Avoid Risks) N N Y Y
Maintenance High Low Low Low
Further Automation N N Y N
Increased Dependency N N N Y

So what will your choice be?

How can i95Dev Help?

i95Dev is a leading ecommerce and integration company in North America. We have products which seamlessly integrate Magento eCommerce with systems Dynamics CRM, Dynamics ERP (GP, AX and NAV) and Dynamics RMS. With our integration products we can help you build the right infrastructure required to truly become an omni-channel business. Our Ecommerce Growth Engine (EGE) product is an integrated ecommerce solution built on Magento platform which gives you access to multiple channels like ecommerce, mobile, social media and online marketplaces while integrating them with Dynamics ERP systems.

Download Now !