Microservices Architecture - Not quite Service Oriented Architecture in new bottle!

Those who are familiar with SOA have often told me ' Microservices Architecture is actually old wine in new bottle! Industry wanted a new Hype and so they came out with concept of Microservices Architecture.' Well not quite so, Microservices Architecture is a subset of Service Oriented Architecture.In fact we can even call Microservices as a SOA design pattern.

Let's take a look at the key similarities and differences of Microservices Architecture  & SOA

The concept of service is common for both the architecture. In both architectures a service has a certain responsibility, a services can be developed in various technology stacks which bring technology diversity into the architecture. In SOA the development of services can be organized within multiple teams, however, each team needs to know about the common communication architecture in SOA. In Microservices Architecture, services can operate and be deployed independently of other services, unlike SOA. So, it is easier to deploy new versions of services frequently or scale a service independently.

                                                                                                                                                            


One hypothetical example (Honestly this rarely happens but we are building a case for microservices) of SOA drawback is since every service in SOA is communicating through ESB, if one of the services slow down, it could cause the ESB to be clogged up with requests for that service. On the other hand, microservices architecture is not designed around ESB and so it has better fault tolerance. For example, if there is a memory leak in one microservice then only that microservice will be affected and other microservices will not be affected.



In both architectures, developers must deal with the complexity of architecture and a distributed system. Developers must implement the inter-service communication mechanism between microservices (if the message queue is used in Microservice architectures) or within ESB and services. In SOA, services share the data storage while each service can have an independent data storage in microservices. Sharing data storage has its pros and cons. for example, the data can be re-used by between all services while it brings dependency and tightly coupling within services.

The main difference between SOA and microservices lies in the size and scope. Microservice is significantly smaller scope than what SOA is and mainly set of small(er) independently deployable service. On the other hand, an SOA can be either a monolith or it can be comprised of multiple microservices.

A Service Oriented Architecture is a software architecture pattern, that promotes reusability of services. The application components provide services to other components via a communications protocol over a network. The communication can involve either simple service or it could involve two or more services coordinating connecting services to each other. Where as Microservices is a software architecture pattern in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. Microservices should be independently deployable, or be able to shut-down a service when is not required in the system and that should not have any impact on other services.  

Where is your eCommerce data saved? How is your data being used? Is your data protected by Data Privacy laws??

Few questions every consumer should ask

1. Where does the e-commerce transaction data & consumers personal data get stored? Does it get stored in India, in USA or some country that provides cheaper storage? How about China or Pakistan? 
2. How is your data being used by the service providers like Mastercard, Visa, Amazon, PayTM? Are these companies sharing insights from your data with other companies?
3. Can Indian governments monitor Data Usage if data is stored in another country? Can government enforce its Data Privacy law when eCommerce companies store data is another country?
4. What happens to your Personal Data in case of hostile situation with the country in which your data is stored?

The Big B of Data - Indian E-commerce Market 2019

The 3 pertinent questions every eCommerce consumer and social media use should ask  - 

1. Where does the e-commerce transaction data & consumers personal data get stored? Does it get stored in India, in USA or some country that provides cheaper storage? How about China or Pakistan? 
2. Can Indian governments keep a watch over Data Usage if it is stored in another country ? 
3. Can government enforce its Data Privacy law when eCommerce companies store data is another country?

Why does India needs Data Localization Law?

Then internet revolution triggered the Data Avalanche and lead to innovations in Data Crunch Processing technologies and Data Analytics technologies. For last 10 years we are creating a totally new type of data that I like to call Event Data or Activity Data. For every activity we are creating 'Data', for example when you move from home to the office,  your mobile device is generating tons of GPS data even when you are not using the device. Data Scientist can analyze this GPS data & extract valuable insights that can be monetized. Today it can be said 'Data is much more valuable than Money! If you want to know why data is more valuable than money in today's world, then read on.

What is personal data?  What is data privacy? What is data localization ? Why should you be aware?

When you swipe your credit card or use Flip-cart or Amazon have you ever wondered how and where your financial transaction and personal information is stored? Well, most of the information ( also called data) is usually partly or completely stored in a database outside India.

For those of you who are not software experts Data is nothing but information like
1) your personal details like don, name, address, phone number, email
2) your net-banking/credit/debit card transaction details,
3) you travel details like ticket history, hotel reservations, cab payments
4) your GPS details on the date of every digital transaction
5) and  many more related details like your bank name, gas company, insurance provider etc
stored in a some computer by the Ebay, Amazon, PayTM, Visa and Master-cards of this world. This data is stored for ever and ever because it is valuable for some company. Over the time you may forget some events but your data which is also called your digital footprint will always be stored in someone's computer.

Apart from worries about who has access to your data if it is stored on overseas computer ( or Cloud which is nothing but a bunch of virtual computer) the Indian government and regulators have limited access to this data across the borders. The RBI wants to change this through its data localization laws and ensure Data is stored within Indian geography so that Indian consumers and government have sovereignty over its data. Data localization is the act of storing data on any device physically present within the borders of a country. As of now, most of the global as well as Indian companies  store the data on a cloud, outside India.

                            Reserve Bank of India's Localization law mandate that companies collecting critical data about Indian consumers must store and process them within the borders of the country. The RBI had issued a circular mandating that payments-related data collected by payments providers must be stored only in India, setting an October 15 2018 deadline for compliance. This covered not only card payment services by Visa and MasterCard but also of companies such as Paytm, WhatsApp and Google which offer electronic or digital payment services.

Why worry about data now?

From the time we started using credit-cards/debit cards/internet banking, the information about your transaction has been stored by card companies , commerce companies & banks in databases and used for 'improving business process and better understanding of the customer'.  There have been instances when some of these companies were found to be misusing the 'consumer information' without consent or knowledge of the customer/consumer. In all probability if the customer was made aware, he would not approve the way companies use his historical transaction information which is protected by privacy laws. This triggered the move by governments of many countries like USA to draft data privacy laws to protect its citizen and their electronic data. 
                               In one of my older posts I had mentioned how your so called harmless creditcard data can be used by marketing companies to predict events in your life and push advertisement to you. One Super market chain was once rumored to have predicted 'pregnancy of its woman consumers based on their digital footprint and they did not even use the social media data' (think about it when you post personal opinion on social media).  The marketing company studied the shopping pattern of one family, their software predicted that the woman was pregnant and started pushing baby product ads to the family, to the shock of the family. (You can read about big data in previous blog post The one who leverages Big Data will win the elections! ) 

It's not just your credit-card and online purchase data that can be misused!

• People started using internet and internet connected devices are creating data 24/7.
• One is creating data when you surf, shop, travel and when you are doing nothing in particular.
• This data gives multidimensional insights and is invaluable to companies as well as governments.
• Analysis of data can tell a lot about individual behavior, choices and habits
• New age software can learn from the historical data and predict behavior and events
• Very few people are aware of Data Privacy.
• Not all data about a person or entity is public data. By law it is illegal to use any data that breaches a persons privacy.

The main intent behind data localization is to protect the personal and financial information of the country’s citizens and residents from foreign surveillance and give local governments and regulators the jurisdiction to call for the data when required. This aspect has gained importance after revelations of social media giant Facebook sharing user data with Cambridge Analytica, which is alleged to have influenced voting outcomes, have led to a global clamor by governments for data localization.

Why data localization is important to a country?

Data localization requires that data created within certain borders stay within them.Data localization is essential to national security because storing of data locally is expected to help law-enforcement agencies to access information that is needed for the detection of a crime or to gather evidence. Where data is not localized, the agencies need to rely on Mutual Legal Assistance Treaties (MLATs) to obtain access, delaying investigations. On-shoring global data could also create domestic jobs and skills in data storage and analytics too, as the Srikrishna report had pointed out. However, maintaining multiple local data centers may entail significant investments in infrastructure and higher costs for global companies, which is why they seem to be up in arms against these rules.At the same time I must say that global companies need to respect every country's right to Data Sovereignty.and understand the risk faced by countries when data is stored in another country. If the consumer data of 1.3 billion Indian falls in wrong hands it can be used to inflict huge damage to the country. sovereignty.

How is Data sovereignty different from Data Localization?

Data localization requires that data created within certain borders stay within them while Data Sovereignty means not only is the data stored in a designated location, but is also subject to the laws of the country in which it is physically stored. Data sovereignty. ensures that data is stored is subject to the legal protections and punishments of that country. So Data Localization is essential for ensuring Data Sovereignty. Russia’s On Personal Data Law (OPD-Law) requires the storage, update and retrieval of data of its citizens to be limited to data center within the Russian Federation.

All of us trust service providers with personal information, both on a voluntary and involuntary basis and we should have greater accountability from these firms about the end-use of this data. Data Localization will ensure that domestic law enforcement can respond more effectively to our complaints.

Robotic Process Automation Hype - Some facts

Robotic Process Automation

This  data was shared by #Pega Systems on webinar. https://www.pega.com/rpahttps://www.pega.com/rpa

Why private and government sector in #India needs to re-evaluate their Data Strategy?

                       

Data Strategy – Time to re-evaluate?

It seems a long time ago that the  3 V’s of volume, variety and velocity was unleashed on the world to describe the evolution of Big Data that organizations were about to see. For years we have been told that we needed to get ready for a new Data Tsunami . We need to be ready to store more data, take data that might not look like we had traditionally from operational systems (such as textual unstructured data) and handle data arriving more quickly.  This was in the web era before mobile and social media took off. Then we had the Big Data storm where all the V’s got bigger, faster and more diverse. When Social Media arrived and the use of external data to help make decisions became a norm the Big Data is everywhere, so much that we seem to have stopped talking about it.

An emerging ecosystem of options

 To deal with big data we needed new ways to store data. This led to the emergence of a new ecosystem of database options to support different needs. New model/schema databases were created with new query approaches to overcome gaps in what was available. Over the time most companies adopted a modified data landscape including NoSQL databases rather than adapting “Hadoop Based Data lake”.What seems to be lacking is a sound understanding of the new COMP-LEXER landscape of data-sources and databases and urgent a need to have a fresh Vision and a new road map for Enterprises Data Strategy.

When Big Data is everywhere, Big Data is just another Data

Today most organizations have stopped thinking about “Big Data” as a challenge that need to be addressed. Now it is just the data that they have to handle to meet different business requirements. Importantly many of those organizations are moving the discussion on to how they get value from that most valuable of assets.  It is no coincidence that focus of enterprises is to get Insights from the data rather than the handling 3Vs of Big Data. It is great that the focus is on deriving value from data. But I wonder if things happening too fast and some enterprise seem to over simplify their database landscape?

Understanding the Complexity Of Data Landscape

The rapid evolution of business requirements has resulted in organizations ending up with an data landscape that has become incredibly complex.  Many organizations are significantly overspending on managing that complex bloated data landscape. The European Data Protection Regulation became applicable on May 25th, 2018 in all member states to harmonize data privacy laws across Europe.Organizations have a huge variety of databases including tabular relational databases, columnar databases, NoSQL databases and the list just goes on.  Organizations have reached this point because they had to meet their business needs. The databases they had were not able to support what they needed to do when they needed to do it.

Tackling the Complexity Of Data Landscape

I believe it is time Organizations should STOP overhauling their data landscape and look for an approach that drives towards a new Data Architecture Vision. It is time to take stock. Think simplification of the data landscape while continuing to meet the business needs today and of the future. Defining a fresh Data Vision and simplification of Data Landscape will help with costs and manageability and help adhere to new Data Protection Laws.  By reducing complexity at source organizations will be better set to use data to create value rather than passing on chaos and complexity to value creators! The evolution of database technologies has been almost as relentless as the progress in other areas of software. Today SQL Server can run on Linux.  Would that make you consider if an open source database is really better than an enterprise grade best in class equivalent you can now use when security and reliability around data is going to underpin everything you do? Look at the fact Graph processing is available in SQL Server and that machine learning capabilities are now pervasive in databases with SQL supporting Python and R.  Would that change the need to create separate data marts for analytics processing reducing complexity and data sprawl?

New deployment options

Finally lets look at the new deployment options.

• Flexible agreements that let you move to the cloud incrementally
• Moving from on premise to  the cloud unchecked lets you reduce the overhead of hardware and having to deal with Capital Expenditure
• Using managed services in the cloud with powerful SLAs to reduce administration overhead while enabling new modes of data storage to support emerging business needs
• Building Hybrid solutions that span into the cloud as needed
• The capability to stand up what you want when you want it and have all that handled with super clear SLAs.

The modern data estate is available on-demand. It spans all deployment modes, offers almost every type of database you might need and helps you find the right ones to meet your business needs. Options abound for simplification, consolidation, modernization and agility within your data landscape all without compromising on meeting your business needs.

Moving forwards

The forwards momentum in database capability and their deployment options  is staggering. Many organizations are not on top of that. Previous decisions, even from as little as 12-18 months ago, can now be revisited to see if your data landscape is running as efficiently as possible.
It is a known fact that progressive organizations, some already because of GDPR, are busy documenting their data assets. In most cases better than ever before. Most of them are focused on what data is where though and how to secure it and ensure it is used appropriately.
Many are not looking at which database it is being stored and if migration and/or consolidation could make life much easier. Be sure to think about your data landscape and consider how it can evolve.
Here are some questions:

1. Have you recently looked at where you are storing your data and do you understand why you have it there? Have you evaluated if there a better option today?
2. Do you know how much it is costing you to manage and maintain your data estate and could reduced complexity reduce that? If lowering IT costs is on your radar this is a sure fire way to find ways to do that.
3. Have you considered if your GDPR compliance would be easier with a less complex environment to manage? Is database consolidation an option you considered on your GDPR journey? If not why not?
4. When did you last evaluate which databases need to be on-premise, which can be deployed in a hybrid mode and which should be able to be totally moved to the cloud? If not recently you may be constraining your potential based on old options and adding additional costs you do not need.
5.  

In Conclusion

A modern data estate will provide options to meet you where you need it to. As you consider your data landscape moving forwards you might want to think about if you are missing a trick by not thinking big picture and looking for vendors who can, perhaps together with partners, cover the entire data estate and all that entails.I have written about need for a Vision & a Road Map for an enterprise and that applies for Data Strategy. as well. The speed at which technologies are evolving and the rate at which new technology get adopted every CTO and CIO should review the Enterprise Data Vision every year and do the necessary change to the Road Map.