Why private and government sector in #India needs to re-evaluate their Data Strategy?

                       

Data Strategy – Time to re-evaluate?

It seems a long time ago that the  3 V’s of volume, variety and velocity was unleashed on the world to describe the evolution of Big Data that organizations were about to see. For years we have been told that we needed to get ready for a new Data Tsunami . We need to be ready to store more data, take data that might not look like we had traditionally from operational systems (such as textual unstructured data) and handle data arriving more quickly.  This was in the web era before mobile and social media took off. Then we had the Big Data storm where all the V’s got bigger, faster and more diverse. When Social Media arrived and the use of external data to help make decisions became a norm the Big Data is everywhere, so much that we seem to have stopped talking about it.

An emerging ecosystem of options

 To deal with big data we needed new ways to store data. This led to the emergence of a new ecosystem of database options to support different needs. New model/schema databases were created with new query approaches to overcome gaps in what was available. Over the time most companies adopted a modified data landscape including NoSQL databases rather than adapting “Hadoop Based Data lake”.What seems to be lacking is a sound understanding of the new COMP-LEXER landscape of data-sources and databases and urgent a need to have a fresh Vision and a new road map for Enterprises Data Strategy.

When Big Data is everywhere, Big Data is just another Data

Today most organizations have stopped thinking about “Big Data” as a challenge that need to be addressed. Now it is just the data that they have to handle to meet different business requirements. Importantly many of those organizations are moving the discussion on to how they get value from that most valuable of assets.  It is no coincidence that focus of enterprises is to get Insights from the data rather than the handling 3Vs of Big Data. It is great that the focus is on deriving value from data. But I wonder if things happening too fast and some enterprise seem to over simplify their database landscape?

Understanding the Complexity Of Data Landscape

The rapid evolution of business requirements has resulted in organizations ending up with an data landscape that has become incredibly complex.  Many organizations are significantly overspending on managing that complex bloated data landscape. The European Data Protection Regulation became applicable on May 25th, 2018 in all member states to harmonize data privacy laws across Europe.Organizations have a huge variety of databases including tabular relational databases, columnar databases, NoSQL databases and the list just goes on.  Organizations have reached this point because they had to meet their business needs. The databases they had were not able to support what they needed to do when they needed to do it.

Tackling the Complexity Of Data Landscape

I believe it is time Organizations should STOP overhauling their data landscape and look for an approach that drives towards a new Data Architecture Vision. It is time to take stock. Think simplification of the data landscape while continuing to meet the business needs today and of the future. Defining a fresh Data Vision and simplification of Data Landscape will help with costs and manageability and help adhere to new Data Protection Laws.  By reducing complexity at source organizations will be better set to use data to create value rather than passing on chaos and complexity to value creators! The evolution of database technologies has been almost as relentless as the progress in other areas of software. Today SQL Server can run on Linux.  Would that make you consider if an open source database is really better than an enterprise grade best in class equivalent you can now use when security and reliability around data is going to underpin everything you do? Look at the fact Graph processing is available in SQL Server and that machine learning capabilities are now pervasive in databases with SQL supporting Python and R.  Would that change the need to create separate data marts for analytics processing reducing complexity and data sprawl?

New deployment options

Finally lets look at the new deployment options.

• Flexible agreements that let you move to the cloud incrementally
• Moving from on premise to  the cloud unchecked lets you reduce the overhead of hardware and having to deal with Capital Expenditure
• Using managed services in the cloud with powerful SLAs to reduce administration overhead while enabling new modes of data storage to support emerging business needs
• Building Hybrid solutions that span into the cloud as needed
• The capability to stand up what you want when you want it and have all that handled with super clear SLAs.

The modern data estate is available on-demand. It spans all deployment modes, offers almost every type of database you might need and helps you find the right ones to meet your business needs. Options abound for simplification, consolidation, modernization and agility within your data landscape all without compromising on meeting your business needs.

Moving forwards

The forwards momentum in database capability and their deployment options  is staggering. Many organizations are not on top of that. Previous decisions, even from as little as 12-18 months ago, can now be revisited to see if your data landscape is running as efficiently as possible.
It is a known fact that progressive organizations, some already because of GDPR, are busy documenting their data assets. In most cases better than ever before. Most of them are focused on what data is where though and how to secure it and ensure it is used appropriately.
Many are not looking at which database it is being stored and if migration and/or consolidation could make life much easier. Be sure to think about your data landscape and consider how it can evolve.
Here are some questions:

1. Have you recently looked at where you are storing your data and do you understand why you have it there? Have you evaluated if there a better option today?
2. Do you know how much it is costing you to manage and maintain your data estate and could reduced complexity reduce that? If lowering IT costs is on your radar this is a sure fire way to find ways to do that.
3. Have you considered if your GDPR compliance would be easier with a less complex environment to manage? Is database consolidation an option you considered on your GDPR journey? If not why not?
4. When did you last evaluate which databases need to be on-premise, which can be deployed in a hybrid mode and which should be able to be totally moved to the cloud? If not recently you may be constraining your potential based on old options and adding additional costs you do not need.
5.  

In Conclusion

A modern data estate will provide options to meet you where you need it to. As you consider your data landscape moving forwards you might want to think about if you are missing a trick by not thinking big picture and looking for vendors who can, perhaps together with partners, cover the entire data estate and all that entails.I have written about need for a Vision & a Road Map for an enterprise and that applies for Data Strategy. as well. The speed at which technologies are evolving and the rate at which new technology get adopted every CTO and CIO should review the Enterprise Data Vision every year and do the necessary change to the Road Map.

Good Read - 3 Simple Habits to Improve Your Critical Thinking By Helen Lee Bouygues

As a Consultant Architect who works with multiple projects in my organization I frequently meet Project Manager to discuss the health of their project. Most of the project managers assure me that their project is doing fine, the client is happy with their delivery and the team members are happy with the leadership only to realize in couple of months that the project got into a problem on all these fronts.  What was the problem ? Surely a executive does not lie to his own organization so how did he not anticipate the problem? Should a manager not be aware of the facts? Problem is lack of Critical Thinking. When leaders accept the facts that are presented to them, when the leaders do not spend time in investigating and verifying the facts..

Many business leaders are simply not reasoning through pressing issues, taking the time to evaluate a topic from all sides. Leaders often jump to the first conclusion, whatever the evidence. Even worse, C-suite leaders will just choose the evidence that supports their prior beliefs. A lack of meta-cognition — or thinking about thinking — is also a major driver, making people simply overconfident. Critical thinking is the process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and evaluating information to reach an answer or conclusion^.

Three simple things to improve your critical thinking

1. Question assumptions
2. Reason through logic
3. Diversify thought

 Do read 3 Simple Habits to Improve Your Critical Thinking by Helen Lee Bouygues

Robotic Process Automation - BPM in new bottle ?

"Is robotic process automation really a new thing or just a new name for Business Process automation?"  Everybody has this question. RPA’s roots can be traced through the evolution of software robotics, so it may look similar to business process management. RPA is really a subset of  BPM and it really focuses on automating mundane human tasks and processes.

BPM (sometimes used interchangeably with business process automation) isn’t a specific piece of software but an approach to streamlining business processes for maximum efficiency and value. It is an in-depth look at how processes are operating, identifying areas for improvement, and building solutions – usually from the ground up. BPM is about making sure the infrastructure of your business processes is solid and decoupled from the implementation  technologies. RPA, on the other hand, is designed to operate processes as a human would, so it exists on a more surface level. It’s faster to implement, ready to use with almost any software, and easily altered or updated to adapt to the changing world. As far as I see it, RPA and BPM are not in conflict with each other. They both have the same goal with different implementation strategies.

While you certainly could use RPA to handle high frequency processes which had previously been performed by humans, perhaps what is really needed is an overhaul of your workflow. If a certain type of transaction makes up the bread-and-butter of your organization’s service, for example, you’ll want to make sure that process is as tight, efficient, and self-contained as possible. There are times when you have to transform the process itself rather than relying on a surface-level fix. That’s is a good case to use BPM.

But transforming a business structure isn’t always feasible. It requires a lot of development and a lot of investment (time and money). You may not have the luxury to build from the ground up. That’s when RPA may be the most fitting solution. If nothing else, you can use RPA to continue operations while investigating a deeper fix.
                   Consider this analogy to self-driving cars: a BPM approach would require us to rip up all paved roads and install infrastructure for the new cars to move about on their own, while an RPA approach seeks to operate an existing car just as a human would. Google's self driven car is addressing  the problem from an RPA angle, because replacing all roads (especially in the U.S.) is just unfathomable. That’s not to say that RPA is always the better option – not at all. The key is knowing the difference and using both tactics to their best advantage.

• Low technical barriers: Programming skills are not necessary to configure a software robot. As a primarily code-free technology, any non-technical staff can use a drag and drop process designer to set up a bot—or even record their own steps to automate a process through a process recorder feature.
• Increased accuracy: Bots are extremely accurate and consistent – they are much less prone to making mistakes or typos than a human worker.
• Meet regulatory compliance standards: Bots only follow the instructions they have been configured to follow and provide an audit trail history for each step. Furthermore, if steps in a particular process need to be reviewed, bots can also play back their past actions. The controlled nature of bot work makes them suited to meeting even the strictest compliance standards.
• No interruption of work: Operations can be performed 24/7 as these bots can work tirelessly and autonomously without requiring staff to manually trigger bots to initiate business processes. If a human does need to intervene, it is to make a decision or resolve an error.
• Existing systems remain in place: Unlike traditional automation initiatives that may require extensive developer resources to integrate across multiple applications, RPA involves no disruption to underlying systems. Robots work across the presentation layer of existing applications just as a person does. This is especially useful for legacy systems, where APIs may not be immediately available, or in situations where organizations do not have the resources to develop a deep level of integration with existing applications.
• Improved employee morale and employee experience: Employees will have more time to invest their talents in more engaging and interesting work. Bots enable workers to offload manual offload tasks like filling out forms, data entry and looking up information from websites, so workers can focus on strategy and revenue-producing activities.
• Increased productivity: Process cycle times are more efficient and can be completed at a faster speed compared with manual process approaches.

I remember automating Java Build and Configuraton management process using Perl scripts 15 yrs back- from verifying that the code is unit tested, checked into version control, code review are completed and then deciding if the code should be pulled into the next build, all was done by automated scripts. We did not call it RPA but if you know what I am talking about you will realize it was nothing but Robotic Process Automation without using a sophisticated tools. If you are keen to know how I automated Java build process, send me a message and  I will post a blog about it.

Why is prototyping important & critical step of software development?

I was working for a client who had bought a software package from one of the ' Market Leading Software Vendor'. The software package  promised to deliver out of the box capability for all his software services requirement out of the box. The Software Vendor'. impressed with  did an impressive demo of  3 of the services  and the client was happy that the product was going to deliver software services in couple of months instead of their development estimates of 12 months to 18 months.The only catch was the 'Software Vendor' told the client that 'most' of the services were ready and 'few' were under development but they would accelerate the development  and try to deliver them them ASAP. The client team accepted the 'Software Vendor' verbal assurance and decided to go with the packaged software (#Mistake) .
          The client bought the software and engaged my company to implement the package software, with a target to complete the implementation in 4 months as per our assumptions & estimates. As the development started we started getting issue with packaged software code, some of the 'out of the box' services were not ready and other had code quality issue - apparently  the code had not been tested.diligently. The 'Software Vendor' told the Client that we should prioritize implementing other services that were ready. The Client was not happy and neither were we as the software implementation team, but Client decided to go ahead and  we started implementing the 'new' services which we were told to implement first by the 'Software Vendor'.

After couple of weeks we realized new set of services were not ready as well and so the client setup a meeting with the 'Software Vendor'. In the meeting we and the client decided to review each of the services that 'Software Vendor' had promised were out of the box and we realized that more than 80% services of the package that was sold to our client was only only on paper and 'work in progress' and they would only be available to us after 12 months. The client threatened the 'Software Vendor' to cancel the purchase and walkout but realized that he has invested quite a lot of money in 6 months that we had worked on the 'Software Vendor' packaged software implementation. It would be suicidal for the client IT team to go back to its management and tell them that they had paid high price for a software from 'Market Leader Software Vendor' without performing a due diligence and after 4 months of work realized that the software was not fully ready! There was no option for the Client team to wait till 'Software Vendor' delivered and push the delivery time lines. The revised budget increased to 1.5X and later to 2X and in spite of this the final product that that Client got was full of bugs (which the 'Software Vendor' promised to fix over in future) and it did not implement some of the services that they had 'purchased 'from the .'Market Leader Software Vendor'.

The moral of the story is that the client made some basic mistakes and did not follow the guidelines of buying a 'Out of the box software package'.  The client obviously was at fault of finalizing the deal with the vendor before contacting my company for implementation. I was not part of the initial meeting with the client so I can't say if my company representatives had advised the client to verify the readiness of the product he was going to buy and whether we had advised a prototyping phase as we should have done.

Why is prototyping important ?

1. Prototypes Help Transmit Intent
Software Packages aren’t always custom fit for your needs. By creating a prototype, the people involved at the earliest stages can better convey their vision for the software and what it’s actually intended to do — and this works better than just describing it through notes.

2. Prototypes Allow for More Customer Involvement
They also allow for that interaction to happen earlier in the design process, when it’s easier to make changes to the software. It’s not uncommon for buyers to ask for one thing, only to realize later on that what they asked for doesn’t work as well in practice as they expected it to… and it’s better for everyone if such problems are found early on. At the same time, prototypes are a good mechanism for explaining what’s technically feasible with the software — and once people know what it can do, they can turn their attention to what they want it to do.

3. Prototypes Provide Users Proper Clarity of Feel for the Software’s Functionality
Related to #2, prototypes serve as a good chance for users to get a feel for what kind of functionality the software will provide once it’s done. Now, at this stage, even the basic functions are far from complete — chances are the software won’t be doing anything more than the simplest tasks. However, that’s still enough to get a good sense of how it’s going to behave when it’s actually done. Without prototyping, the software could end up feeling wrong to the users — and that doesn’t help productivity.

Here is the right way to invest in a 'Out Of the Box Software Package Product' and this applies to every industry vertical.

1. When you buy package software try not to be the 1st implementer of the product is my 1st advice
2. Don't get impressed with vendor's demo of few services of packaged software. Demand a complete demo of entire implementation of the software services that you are going to buy deployed on a similar software environment and configuration as yours
3. Form a team of technical and domain analyst to explore the demo environment, create a review checklist that covers all critical aspects , review each service in the software package and submit a detailed review report
4. Make sure the software package implementation can be customized as per your requirement
5. Look at the code of the software and so a sample review of the code quality - 'Software Vendor' should not have a problem in allowing you do review the code you are investing in! 
6. Finally insist on a prototyping phase and implement few critical services on a replica of your production setup and test the prototype once it is implemented. You can also do a round of performance testing on the prototype
7. Make sure the users in your enterprise are involved in all the above stages and they provide their feedback.
8. Once  above steps are completed with satisfactory result you can go ahead and invest in the 'Software Vendor'

The above checks apply to almost any software small or large that you invest in. I hope sharing my experience will be helpful to enterprise & people buying 'Vendor Software Packages'