Does RDBMS tick all the check-boxes of a blockchain database for your system?

We know that Blockchain stores information in one immutable structure. Reliability, availability, and suitability are implicit characteristics of this decentralized database and multiple copies of data remain synchronized.
                         Just as cloud-based application services provide common functionality that allow developers to focus on key features to realize their applications, blockchain database incorporates data distribution, replication, synchronization, and update facilities, eliminating the need for developers to implement these features into each application. Storing all information in one place, allows for deploying a single suite of audit tools for reporting on all financial transactions as they are enriched by numerous applications during their processing.

Enterprises have been using  RDBMS like IBM DB2 and Oracle and would like to avoid another technology called blockchain if they can implement the same functionality & features using a existing database technologies. We know the strengths and weakness of RDBMS and we need to see how RDBMS can fit the blockchain use-case. If we have to use a database to implement say a Bitcoin use case where participating entities will either Read/Write and there is no requirement for updating or deleting rows from database then we can expect as very high improvement in the performance of a RDBMS.  Now add feature of partitioning to the database where you have option to partition a database by various parameters like transaction scope, day/week/month, partner_id and the resulting database will have even better performance for read and writes operations.
                                    So question to ask is why not implement a RDBMS database by designing it for blockchain use-case instead of investing in blockchain technology which requires additional investment and resources in terms of hardware, maintenance, security.

A partitioned distributed database system can perform the job of a Blockchain ledger and can be designed to give comparable or better performance than blockchain along with better reliability, availability and scalability. The following table highlights some key considerations that I could think of when I design a RDBMS database to support a Blockchain use-case

Lets take an automobile industry use-case. 

A car has a manufacturing defect that results in a part being replaced. The typical communication chain could follow the following pattern:

• The car owner brings the car to the dealer to diagnose a problem

• Car dealer inspects the car and notifies the manufacturer about the faulty part

• Manufacturer works with the part supplier to determine where the fault lies

• Part supplier and manufacturer agree that the part is faulty, then notify the dealer

• Car dealer notifies the customer
• Car owner brings the car back to the car dealer for part replacement

In this scenario, communication among the participants customer, car dealer, manufacturer, and part supplier can be delayed by incomplete information, so full and accurate responses along the chain are impossible. Second, all communication goes through email, telephone, or postal mail, and the customer has to visit the car dealership multiple times to have the part replaced, possibly being unable to use the vehicle until the car is repaired. This is a very shabby customer experience.

Now let’s look at the customer experience when blockchain style technology platform is implemented. In the same scenario, the communication chain will be as follows:
                                                                     

• The IoT sensor automatically notifies the manufacturer, car dealer, and car owner about the defective part

• Car dealer contacts the car owner and sets a service appointment for an inspection

• Car owner brings car to the dealer for inspection, which confirms the faulty part

• By now the information about the faulty part is already put on blockchain, which has by now notified all the parties – manufacturer, insurer, part supplier, car dealer, and owner

• Manufacturer, dealer, and part supplier collaborate to analyze the car’s IoT sensors and dealer inspection report to identify where the fault lies

• Manufacturer gives an instant approval for part replacement on blockchain, which automatically notifies all parties

• Car dealer replaces the part and delivers the car back to the owner

When can we use Database with Blockchain design?

When all the participants of the transaction are known entities and registered in the system, Blockchain's universal availability feature is not going to be particularly useful here. What is useful from Blockchain is the concept of the immutable ledger , availability and notification to participants and it can be achieved by a traditional RDBMS.

Big Data Reference Architecture For Telco

>

Value from Big Data at Rest & Big Data In Motion

Big Data is of value when we process it for insight but did you know Big Data in motion is as important if not more important than data at rest. Enterprise processing big Data in motion have a edge over competition because they are deriving intelligence and insight from data  in motion even before it hits the database, using technologies like Complex Event Processing . A diagram to explain how value is derived from Big Data At Rest & Big Data In Motion.

How can Indian Army leverage Artificial Intelligence in today's Big Data World?

It is 2025.

Indian Intelligence has petabytes of unclassified social media posts. Machine learning software isolate images of potential Criminal elements agitating protests, cross referencing cell phone pictures are posted on social media with traffic cameras, and more sensitive collection platforms.

Intelligence agency distribute the images of Criminal elements on a cellular alert network that lets concerned citizens turn their cell phones and other personal devices into a civil defense sensor network. This civil defense network acts as a cloud, helping cyber defense apps secure critical infrastructure and conducting predictive models of where possible cross-border insertions might occur based on historical data, weather, terrain, and news reports.

The technology in this future battlefield is already driving a wide range of commercial applications. From Amazon figuring out what book you want to buy next  to Google optimizing the ads you see while searching, we live in a world defined by “big data” and artificial intelligence applications that identify patterns in our consumer habits and daily life. These applications have the potential to change the character of warfare. The countries that adapts accordingly and integrates artificial intelligence across the force will have a generational advantage on the battlefield. 

                                           The NIA & Indian army needs to develop a strategy for integrating narrow artificial intelligence applications into the force. Existing investments in artificial intelligence of most countries tend to emphasize future autonomous systems such as tanks, robot soldiers, and planes that can operate with minimal human input. An alternative approach is to experiment with predictive models and big data to increase the combat power of the current force and that can only be done by building a sensory network by setting sensors at strategic locations as well as using the mobile devices of citizen as virtual sensor network.

What is Artificial Intelligence?

Artificial intelligence is commonly defined as the theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making & translation between languages. It can range from weak forms, such as narrow artificial intelligence, that processes big data to answer basic questions and generate predictions (e.g., Google maps helping you drive home) to strong forms such as “Artificial General Intelligence” and “Artificial Super Intelligence” that exceed human intelligence, creativity, and adaptability. Short of the promise of driver-less cars and robot servants, narrow forms of artificial intelligence like machine learning are starting to change sectors ranging from healthcare to logistics. With respect to public health, the flowminder.com uses narrow artificial intelligence to predict the spread of diseases. In logistics, machine learning is helping companies make supply chain adjustments, optimize delivery routes, and design warehouse systems. Vehicle manufacturer are applying artificial intelligence to improve predictive maintenance of motors. Volvo collects data with smart sensors on their vehicles and applies machine learning techniques to conduct diagnostics that reduce down time for services and better inform the resupply.

Artificial Intelligence for the defense forces

While many commercial applications of artificial intelligence are based on identifying patterns and trends using big data, most military applications focus on autonomous systems.  Example of an artificial intelligence programs in the US governments unmanned undersea and aerial vehicle programs such as LOCOUST (Low-Cost Unmanned Aerial Vehicle Swarming Technology) & GREMLIN anti-surface-to-air missile drone program.  Another ideas is for the army to leverage existing unmanned vehicle systems and develop a logistic system that can be used in combat scenario where multiple unmanned vehicles can intelligently follow a manned vehicle.Technology is being used for civilian services and it needs to be adapted by Intelligence  agencies and armed forces. Border and Marine border security can be improved with reduction in human patrol by using Big Data & Predictive Data Analysis. Are armed forces & intelligence agencies already using these technologies?

x

Could we not have implemented Aadhar System in an inclusive manner? (Part-4)

Dear decision makers,
The first thought that came to my mind when I heard of  new Aadhar Face Recognition feature was that, next time Jio might refuse me a SIM if my face recognition does not get validated! What was originally a scheme to ensure that social benefits reached the deserving has now become a number in Jail Roster or so people fear!

The infamous Government Babu's of India who so far were famous for quite a few wrong reason are now custodian of my personal identity data including bio-metrics and iris scan. A SIM card is a commodity like any other commodity and it is definitely not a government benefit for which we have to validate our bio-metrics. We pay for the SIM service and it is responsibility of telecom company to validate I am who I claim to be. So what is governments interest in offering private Telcos to use the bio-metrics validation which was originally meant for Social Security distribution? Mr Modi is friend with quite a few world leaders. Will he care to ask the heads of USA ,Australia, Israel,Germany, China or Japan if they take the responsibility of validating SIM users identify for the private telco companies? So why is India so keen to do the identify verification job for Jio ? And now the privilege is extended to every telco company! I can understand that government wants to prevent criminals from using SIM using fake identity documents but using bio-metrics and face recognition to buy a SIM card is stretching things too far. Next the grocer may refuse to sell sugar if I do not have Aadhar card , quite like a woman was refused medical treatment in a hospital and died few days back.

Did government ever wonder why do USA, UK, Singapore did not capture bio-metrics of their citizen for their UID program? Do these countries have more terrorism incidents or higher crime because they do not use  bio-metrics like #Aadhar? How is Indian smarter than these countries who have Avoided Risking their  Citizens Privacy and Identity by not Capturing their iris and bio-metrics scan?

Who is making these decisions about #Aadhar for the country? Is it the Indian PM? or the Home Minister? or Mr Nandan Nilekani? From when did India start implementing national policy decisions made by a select few people without review by all the 750+ Member of Parliaments, who are true representatives of people in the parliament? Are the MPs not capable of conveying people views and concerns? Or are we no longer government of the people, for the people and by the people? Finally why is the Supreme Court not asking the government to include MPs in such a major decision making process that effects the country? Do the oppositions MP becomes unworthy of opinion because their party did not form the government? So may question that no one dares to ask because most people don''t know the impact of data leak of Aadhar data and the future of India is stake if identity of Billion people is risked even once. Guess it is too late already as the data seems to have leaked too often or else The Tribune reporter would have been in Jail  Chakki Peesing & Peesing & Peesing as Dharmendra said in Sholay)

We expect some maturity from the individuals who form the government irrespective of which party they belong to. I assume government has not forgotten that their term is only 5 years unless they win again and if they lose next election then the opposition parties who are objecting to Aadhar policies will come to power and change the policies again.  Is this not waste of national resources? Is it not duty of an elected government to forma  committee to get views of all the elected representatives and implement majority view & wish before implementing a software system that has potential to screw every Indians happiness?  So why did government not form an all party committee to decide the vision for Aadhar? As of today no 2 political parties agree to governments Aadhar vision in totality so every time the government changes can we afford to change Aadhar Software implementation?  I am also shocked that opposition parties not demanding that every elected peoples representative in parliament should be included in this decision making on Aadhar, the 1st of its kind scheme which is literally being forced upon people?  Another surprising fact is that the Supreme Court that has taken suo moto decisions on various cases of less importance does not feel that it should direct the government to form an All Party Aadhar Decision Making Committee ?  Gives me the feeling that for most fo the wise men of India ignorance seems to be the bliss An immature nation, barely exposed to IOT and the risks of internet are living in blissful ignorance.

I have asked enough questions, now let me talk about the right way to implement a software that can change the course of a nation. If I was the Prime Minister of this country, responsible for the actions of his ministers and his government I would be bloody concerned about the Aadhar project. I am not the type of man who will let the team do the work and when they screw-up ,tell that we screwed up because the Ministers did not do their homework or that the IT team was not smart to highlight the risks and mitigation.Let's be clear, Success or failure the onus of this ambitious scheme lies with the head of the government. So to implement a scheme like Aadhar my government would do the following things-

1) Aadhar is a scheme for the benefit of people, its a scheme which takes responsibility of capturing and protecting database of identify of every Indian and that is why it cannot be implemented without having all elected representatives of people on board. Period. If I can't convince 790 member of parliament (MP) of Indian parliament (250 Rajya Sabha & 545  Lok Sabha) that Aadhar is beneficial and safe for Indians, how do you expect me to convince entire country?

2) Form an Aadhar Vision team of MPs, ensuring each state & union is represented by at least 2 MP lead by the Minister of Home and ask them to draft a vision for Aadhar card along with help of IT, security and policy experts.

3) Form a review committee of another set of  2 MP  from each state & union to review the Vision document created by the Aadhar Vision team and finalise the 1st draft for review by Home Ministry.

4) Publish the Vision on the website and take inputs from the public, God knows we have enough smart people in India to give create inputs and this is the age of crowd sourcing so why not use the knowledge bank?

5) Final Aadhar Draft is reviewed and presented in parliament with all the above inputs and gets approved by the parliament for implementation.

6) The Aadhar Vision is given to the Aadhar Information Architect to define the solution architecture and road map for the Aadhar System. The Solution architecture is reviewed by 2 independent IT consulting firms who are bound by a NDA (non disclosure agreement) and the final draft of solution is ready for implementation.

People may say that 790 people can never agree to a common vision but they would be wrong. GST was implemented including each Indian state and union territory and thousands of people were included in the decision making process and here we are talking about only 790 people who represent the entire country. Aadhar cannot be solution to all problems so let's not cook the hen that lays golden egg! My final word for those leaders who think Digital is magic and Digital is future. Let me be burst your bubble about the New Digital India! Indian digital journey started in 1980s when  Rajiv Gandhi & Sam Pritroda revolutionized Indian telecom sector and STD booth started poping up everywhere. People started using the squeaky 56kbps modems with their phone lines to access internet, the state governments started their digital journey way back in 2001 with their e-governance initiatives. I had the privilege to be part of a team that bid for Goa state governments e-governance project in 2001. What has happened in last 2 years is there is lots of marketing and that has brought Digital to focus. Even the IT companies are MAKING HAY as the Digital sun shines. IT companies don't want to tell their customers that they there is nothing new about Digital except the renewed focus. Let me also remind you that many large enterprises have burnt hands when playing with Digital because they jumped into the digital well without a clear vision and road map. Problem is that with Aadhar, government is not jumping in the well alone,  they are taking the plunge into the well with 1 billion people and risking their lives as well. So decision makers please keep your ego in  your hip pocket and spare a thought for India. Do you really require government employees to use Aadhar to validate their daily attendance as it is happening in Mumbai Corporation and will soon be implemented across India? How is this stupid implementation ánd many similar usecases going to be useful for India''s progress and Digital journey? Sorry Mr Government you have made a blunder by now consulting experts and not learning from other countries like USA, Canada, Japan and UK. Correct your mistakes before something worse happens because history never forgives anyone. Jai hind!

Where there a genuine will & concern for people,

then pray, one will always find a way. 

Whre there is ego, over confidence & arrogance,

then failure is only a step away.

Your may be interested in my past posts on Aadhar

1) Why you should be concerned about #Aadhar being made mandatory for citizen of India?

2) Whats is wrong with Aadhar Software System ? (Part-1)

3) Whats needs to be fixed in Aadhar Software System ? (Part-2)

Whats needs to be fixed in Aadhar Software System ? (Part-2)

If you would like to read my past post on #Aadhar here is the link  Whats is wrong with Aadhar Software System ? (Part-1)  You may also be interested in reading a previous comparison between SSN & Aadhar at this link Why you should be concerned about #Aadhar being made mandatory for citizen of India?

Continuing from  

Whats needs to be fixed in Aadhar Software System ? (Part-1)

One pertinent question I want to ask UIDAI is, for a company that in implementing Aadhar should it not be a prerequisite for it's employee & service partners to have an Aadhar number and use Aadhar for registering new user of Aadhar System? In this alleged data leak case how did the system allow new ASA and AUA users (Authentication Service Agencies & Authentication User Agencies ) to be added without their Aadhar numner keyed in? Would the alleged hacker dare to sell data if he knew his activity was being tracked by his Aadhar number and that he would be caught? Software Design flaw eh?

I found the so called Aadhar System's Vision at their website and it is the briefest software vision document that I have seen in 20 years. Even after reading the Vision you won't know what are all Services that Aadhar is designed to provide today & in future. Our understanding was that Aadhar would be used for distribution of social security benefits to the eligible. Now goverment wants to make Aadhar the Single Source of Truth of your identity, something even the USA & UK do not dare to do with their SSN because of their concern for the privacy of their citizen. You have to read my earlier post to know why USA does not use bio-metrics for their SSN which is similar to Aadhar (Link)  Now your Aadhar number will get verified when you open a Bank accounts, do Stock investments, buy insurance and even when you get admitted to a hospital? Why are we using Aadhar for the financial transaction tracking? Was PAN not supposed to be the 'Universal Identification for Financial Transaction' as per the Income Tax website? I can understand that  goverment wants to link PAN and Aadhar to ensure people do not  create multiple PAN  and that makes sense. But when you are linking PAN with Aadhar where is the need to provide Aadhar to Bank & Stock exchange? They already have my PAN which is already linked to Aadhar (Read older post Linking Aadhar )! This is just bad,bad, bad, software design,  Last week it was in the news that a lady who was ill tragically died because she was denied admission to hospital because she could not produce Aadhar card. Who has given this ABSURD instruction to hospital that Aadhar is required to get medical treatment or to get admitted to hospital? Even the Nazi did not have such inhuman laws! Recently someone raised a valid question - How many homeless and nomads people live in India? Without a address how will they get Aadhar card to receive social benefit? I would have liked to know if such problems were already thought by the UIDAI ? Are these problems part of Vision document and is the Vision shared somewhere on their website so citizen can read and be aware of Aadhar Features.

If you want to build a Digital India start educating people and making them aware of what is coming their way. Sadly goverment schemes are very poor at educating citizen and GST is another example. Govt announces GST & the PVSindus 1 minute TV commercial does not tell how GST is going work. Who is going to educate the business men? Why could goverment not conduct 1 hour TV Training on all TV channels to demonstrate and educate business community on GST? When you go shopping talk to your grocer and you will be shocked to know that 50% of the times your grocer does not know how and when GST has to be filed! Sorry Sir, you cannot go Digital without training the last man & woman in the village, in a medium & language they understands. Everybody is not as smart as people sitting in Delhi!

Software engineering works on well defined Software Development Life Cycle Process. Software Managers help client define their Software  Vision. A Software Vision document defines the high-level scope and purpose of a program. It is a clear statement of the problem, proposed solution, and the high-level features of a product helps establish expectations and reduce risks. So a vision document is kind of a Geeta or Bible that has all major functionality and behavior that has to be built into the software to support the enterprise's business goals. If something is not in the vision it will not be part of the software road map & software design. Period. Next we create a Software Road-map which has milestones for different functionality that has to be built into the software and then the software architecture and design work starts. What I want to highlight to non-IT folks is that if you don't have a documented vision or if your Software Vision changes every year/months than it requires frequent changes to design, leading to patchy software and it affects the software quality including quality of software security.
                I believe Changing Requirements are a big challenge is software that we develop for government. The government changes, the Vision changes and software has to be changed - no scope for arguments! Take Obama Care where USA has spent millions on the program and developing software for the online program which might get scrapped now or else take Indian Aadhar Card. Vision of the government that started the UIDAI program was at least 90 degree if not 180 degree apart, from the vision of next government. Now think about different Government Software that have been built by past government, how subsequent governments have changed the Vision to what software should do and how software would have gone through structural changes making it patchy.

To give an example to those who are not from IT, imagine we order custom built Luxury Sedan, then goverment changes and wants the sedan to be modified to work like a Bus. Again the goverment changes and they want the vehicle to be again modified to work as a Bullet Train!  The chassis of the car can only take load of 5 to 6 people and you change the requirement to carry 60 people or worse 1000 people? Don't you think it is Better, Cheaper and Safer to use the car as it is and build a new Bullet Train from scratch? That's what happens to a software that starts as a modest application to perform few services and then client gets ambitions and wants to continue to modify the same software to serve the entire humanity. If the software foundation was not meant to handle all the ritz then it is better to start from scratch and build a new software. I hope that's not what is happening to Aadhar but it could if the people who make the decisions are not advised about the impact by their software architects.
                       I believe it is a duty of an IT engineer (and every professional ) to highlight the risks to the management on paper and provide the best recommendation to implement the software system but never compromise on quality and security of a software. I wonder if Aadhar software architect & engineers have done their duty well because the operations issues like data leaks and publishing data Aadhar data on their website do not give me the confidence that it is a robust system..

In my 1st post I mentioned Aadhar is currently being used for taking attendance of municipal employees and I think that is WRONG use case for Aadhar and if I may say so it amounts to abuse of a software system. Hey, I want to use Aadhar bio-metrics verification for my driver & housemaid to make sure she comes on time. Can I please have it? There was news that children will have to provide Aadhar to get admission in school and I am not sure that is makes sense to implement that because one study says that bio-metrics of children change quite frequently and it will create issues if bio-metrics cannot be verified. So much for basics of software design now lets look at it as a black box system and visualize an 'ideal software architecture' that can support services that we know Aadhar is going to be linked with in future.

Aadhar currently is an identify proof that will confirm that you are you and you don't use a fake identify to fool the government services like Passport or Income Tax department. Aadhar system should provide an web page to allow various departments to verify your identity by entering your Aadhar number & getting a validation response from UIDAI server. It is a basic validation service that guarantees that you are who you claim to be verifying Aadhar number and maybe asking you a random question about your Aadhar data like DOB, COB etc.  USA uses SSN for validation and the below image  is the response message that you get from SSN validation  no chance of anyone viewing your SSN details. So why do some uses have access to download users Aadhar Data ? Why does the Aadhar system design allow a use access to data? What UIDAI has to understand is it is a poor design that you are providing a interface for someone to download  Aadhar data. Period. So don't pass the blame to user if he downloaded and sold the data because your design itself is faulty.

If the Aadhar system allowss sharing your Aadhar details with private concerns like bank or letting them access your profile using your Aadhar number then it is a huge risk to the security and privacy of an individual and that is why people have filed PIL in court. What Aadhar system should do is allow a user to enter his Aadhar number and in response show him a standard success or failure message like the Americans show for their SSN!

Moving on lets discuss a model architecture and its key components that would ensure that we have a well designed that will work smoothly. What are the check points of making the system secure, fool proof, intelligent & proactive? How to build  software a system that notify authorities when a miscreant tries to access restricted data using password or even if he manages to hacks into the system bypassing authentication (if hackers can get into Pentagon you should be prepared for the worst)? That's what is coming next, right here on this post, in another couple of days. I will leave you with a diagram of a model architecture for now.

Sample block architecture for an enterprise system

Thanks for visiting my blog.
                                                 

Speaking Technically - My take on Aadhar System

Some time photos express better than 1000 words. So this is my attempt at creative art!

Whats is wrong with Aadhar Software System ? (Part-1)

I can't say I was shocked to hear about The Tribune (link) report on how their reporter purchased Aadhar data for 500 INR and for another 300 INR purchased a software that could print Aadhar Card ( India's Social Security Number card) but I was surprised at the poor design and operating model of the system. I expected the system to be much more smarter, secure and foolproof than it seems to be now. UIDAI has denied the data leak but Tribune seems to have material evidence and they have published a response to UIDAI statement today (link).

This kind of data leak does not happen because of  a single point of failure. There has to be issue with the
1) Solution architecture (architectural description of a specific solution)
2) Application architecture (describes the behavior of applications),
3) Security architecture (unified security design that addresses the necessities & potential risks),
4) Operation architecture (defines control of operational procedure & execution of operational tasks)
5) Finally the Software testing plan seems to have failed to detect the flaws in the software.

As long as there is valuable data, hackers will try to attack the system but fortunately this was not a hack or else hackers would have wiped the data and held Indian government for ransom. Politicians who are enforcing Aadhar just don't seem to understand a badly designed software can play havoc and that delayed software system is better than a faulty software system. I don't think we need any more proof to call for an immediate software audit conducted by 2 independent reputed agencies. If we are so sure that system has no flaw then software audit will help regain the confidence of  billion Indians. Every big software is audited periodically so why not audit Aadhar System which is a critical system for the government as well as billion Indians. I don't understand why government has not initiated a software audit after multiple 'reported' incidents of data lapses over last 2 years. If the audit highlights any issue then the Aadhar software architects are to be blamed and not the government. The other issue could be the change in Vision, possibly Aadhar system is made to do what it was not designed to do when it designed.. A software audit will only validate that you have a well designed, robust and secure software system and its operations are being managed well by the team.

When  there are multiple issues in a software system they can only be addressed by conducting a detailed software audit followed by fixing the issues identified in the audit. ( Do read my earlier post  Why you should be concerned about #Aadhar being made mandatory for citizen of India? )  Having worked as a fire fighter on solving complex issues on many complex software systems for fortune 100 companies I must say I have not encountered such a large software system that seems to have been put in production without due diligence. I have seen projects where  leads & managers have been removed on the spot for much smaller issues than what was reported by The Tribune. Software systems are custom made to solve a particular business challenge and if there is a failure in basic operation of the software then it is because the project leadership was incompetent to handle the project right from the solution design stage to development stage and onto the operations stage.

Here is a simple picture that will give you an idea of what all could have gone wrong with Aadhar System, I am not privy to the Aadhar architecture so the picture highlights the potential issues in the software based on what we know from reports on data leaks. I am concerned that a software system has become a prestige issue and UIDAI is defending reports rather than publishing a whitepaper on health of the Aadhar System so Indians don't worry about the most critical software that is going to affect every Indian. I want to find out , what is wrong with Aadhar system? What needs to be fixed to avoid the known issues in future?

My understanding of Aadhar System

< Click image to zoom >

So what are the key take away from this data leak incident?

1)  Critical software system like Aadhar are used for verification of Aadhar user's identity and when user input is passed to the system, the systems responds as verification success or failure. The system should not allow users to download Aadhar number or details of Aadhar users. This is a grave software design flaw and there is no doubt about it whatever UIDAI might claim.

2) The data access should have been restricted at application service layer, data service layer as well as database level for additional security and I think all 3 layers have design flaws.

2) The system should have different levels of user authorization that limits user access to certain services and this authorization framework seems to be flawed or else missing from the system.

3) Creation of new user is an activity that should be restricted to a limited set of super users & if we go by the news report then it seems that a use who is not administrator has the right to create new users without approval of any superior authority and this is poor design and a big security risk. A new user creation process is typically performed by the administrator and for critical systems like Aadhar, there should be an 'approval workflow for user creation' where clearance is taken from 'access control board'  or at least a super admin before creating new users. A ordinary system user who accesses the system for routine transaction should not be allowed to create new user at his will. This flaw allows the system to be misused as it seems to have happened in this case.

4)  Authorities have responded that system activities are monitored so it is surprising that when users are being created at will without approval no one noticed it, when system was being accessed by unknown new users no one noticed it, when user data was being downloaded no one noticed it and even the network administrator did not notice unusual activity on the network when billion records were being accessed/downloaded.

                                                                         

< Click image to zoom >

5) In an ideal system that has Business Activity Monitoring, rules are defined on what services needs to be monitored, what alert should be sent by email or by SMS in case of some user accessing a set of services and what action should be taken by the team who is responsible for monitoring the software. None of these things seem to be in place or else UIDAI would have nabbed the culprits before the whistle blower initially complained to authorities and long before reporter investigated the story.

As I mentioned earlier this looks like case of multi point failure and if one of the checks had failed still the monitoring system should have automatically notified the authorities about the suspicious activities within minutes if not seconds of the suspicious event. My guess is a good Business Activity Monitoring tool which is must for any critical enterprise system is either not implemented or the implementation is flawed.

Obviously there could be other issues in Aadhar system that we are not aware of. For example scalability and availability issues and performance issues - and I have reason to believe that  the way it is being implemented today Aadhar system is going face major performance issues one day. The 1 lakh plus municipal employee in Mumbai are using Aadhar bio-metrics to sign-in & sign-out at work.  (I have got proof that municipal employees get emails twice a day from Aadhar system when they scan their finger prints). Which means Aadhar system is being accessed for employee attendance 2 lakh times every day by Mumbai municipality alone! There are around 4000 cities in India which makes 4000 x 2 lakh = 800,000,000 hits to Aadhar server everyday by municipal employees alone! I assume if municipality is using Aadhar for attendance then other government employees will also be using Aadhar verification for attendance and a conservative guess would be 1 lakh government employees across 4000+ cities in India ( I am not considering the employees working in smaller town & remote areas). This means there will be 1.6 billion hits to Aadhar server everyday just verification of attendance of government & municipal employees!  At 9 am when all employees reach office  there will be at least 80,000 hits to the server every 3 second (assuming 4000  cities , 10 offices in each city, 2 bio-metrics scanner in each office)  Did UIDAI plan for this? Is Aadhar Architecture built to take this kind of peak load?  Why the heck should we use national  bio-metrics server to verify that employee in each city has reached office! Did we create Aadhar for such stupid mundane tasks?  Which other SMART DIGITAL COUNTRY in the world has implemented this kind of verification system for government employee?  Which software architect gave this idea to goverment to use Aadhar for employee attendance verification and what are other redundant uses that Aadhar is going to be used for?  It is absurdly, insanely, mind boggling crazy to implement Aadhar based attendance! Imagine when Hospitals, Railways, Airlines, Jios  & every other company starts using Aadhar for verification we will need a new Aadhar Hardware City to host the servers required to cater to such large population! (Ok! Ok! I exaggerated it! Wont need a new city but a huge number of computing nodes on a Cloud since it is quite likely that Aadhar system is hosted on a Cloud)

In my next post I will explain the potential missing blocks that could lead to such system failure in a software system (again I am assuming The Tribune report is reliable and they have indeed purchased billion records for 500 INR. Damit!)

My next post will be about Whats needs to be fixed in Aadhar Software System ? (Part-2) & after that 3rd post will be on How blockchain or similar trust framework could have prevented the Aadhar Data leaks in Aadhar System? (part-3)

Is blockchain viable for a Business Process Management solution?

When we talk to people about Blockchain we realize people seem to have various perceptions that is formed in the context of Cryptographic Currency and since the technology is incubation I guess each perception has to be respected. After 19 years in the industry working for fortune 100 clients I have acquired habit of doing postmortem of each new technology and try to predict the acceptance of the technology. I have been proven right on many occasions and not so right on few occasions.

I agree with people who looks at Blockchain an interesting technology advance that may solve some challenges (and create new challenges) but I don't agree that blockchain is he biggest thing since Internet as some people claim, in fact it is far from maturity and at the most it is a good prototype according to me. Taking a step back.

What is blockchain?

A block is the ‘Current’ part of a blockchain which records all of the recent transactions, and once completed goes into the 'Chain of blocks' as permanent data record. Each time a block gets completed, a new block is generated. There is a countless number of such blocks in the blockchain. The blocks are linked to each other like a chain in proper linear, chronological order with every block containing a hash of the previous block.

 And what is blockchain in Bitcoin context?

A blockchain is a public ledger of all Bitcoin transactions that have ever been executed. It is constantly growing as ‘completed’ blocks are added to it with a new set of recordings. The blocks are added to the blockchain in a linear, chronological order. Each node (computer connected to the Bitcoin network using a client that performs the task of validating and relaying transactions) gets a copy of the blockchain, which gets downloaded automatically upon joining the Bitcoin network. The blockchain has complete information about the addresses and their balances right from the genesis block to the most recently completed block.

So whats great about Blockchain in the BPM context? 

Have you implemented workflow solution where a work-item has to pass through different people who work on it in a sequence before the work-item is processed and work is flagged as complete? Or maybe implemented a BPM solution where a business process has multiple tasks and has interface with multiple internal systems for validation and reference data which makes the process slow and consumes resources. What if each work-item in a business process was 'Smart Work Item' and could carry all the data references required to process the work item so that network IO was minimum? The processing would be faster, we would eliminate wait times and dependency on related tasks and business would run with optimum efficiency. I am assuming you are familiar with BPM so I wont give examples to explain the above statements. This can be achieved by a distributed ledger  (as in Bitcoin Blockchain) or a Smart Business Object (as I like to call it) that is accessible only to the authorized parties across the internal or external network.  The Smart Business Objects (SBO) are encrypted data objects with a private and public key and they can be viewed, updated by the authorized parties but they cannot be deleted by anyone and at the end of business process flow/s, when the life of the SBO is over the object would still be available on the internal network in view only mode.

We implemented something called as a BPM Workflow Reference Database for one of our clients ( I concede that we were not smart enough to think of a distributed ledger) . So adding a new database that kept a track of state of the work-item by referencing it with a workitem-key helped reduce the network calls from the business process to a maximum of one call to get the state of the input reference data at each step of the process and this optimized our solution by a huge margin but we had to create some database triggers and batch processes to keep the Reference Data updated.
What do enterprises achieve by implementing business process management?

• Operational Efficiency – Streamlined business processes lead to streamlined operations, with greater visibility and control. The automation of repetitive, low value tasks can also lead to cost savings.
• Compliance – All business processes are carried out in-line with a company’s procedures and process documentation can be generated for demonstrating compliance in an audit.
• Competitive Advantage– A BPM solution ensures that processes drive operational efficiency, business visibility and can quickly adapt to customer requirements, which helps to provide a strong edge over the competition and shorter time to market
• Scalability – Automated processes scale much better than manual processes – ideal if an organisation is expanding.
• Agility – A BPM solution will enable a company to quickly update its processes in response to developments in its operational environment.

Great! So we have implemented BPM for years and clients have benefited immensely but there are some challenges that create bottleneck in BPM solutions and blockchain can solve them. But how about using a traditional centralized database to perform like a distributed ledger? I am not suggesting we deploy a database at each node rather we add a independent database that acts as Ledger Database & can be accessed by all trusted systems over the network. The Ledger Database should have add-on features of blockchain namely immutability, no central authority & global availability of asset over network. Unlike crypto-currencies a BPM solution does not have unknown participants so I think it should work well.

Disadvantages of using a Blockchain / Distributed Ledger - 

The blockchain as a database is not so great, measured by traditional database standards: throughput is just a few transactions per second, latency before a single confirmed write is 10 minutes, and capacity is a few dozen GB. Furthermore, adding nodes causes more problems with a doubling of nodes, network traffic quadruples with no improvement in throughput, latency, or capacity.  Worst thing is that Plus, the blockchain essentially has no querying abilities.

Advantages of using a Ledger Database-

• Globally accessible database to store data & documents
• High capacity and throughput for millions of records and documents
• Interchangeability to store large files and media
• Data immutability that brings trust and audit-ability to the records
• Query technology that enables quick retrieval records and documents
• Reduced security and liability in managing data

There is need to prototype and compare the 2 solutions, one with blockchain ledger and another with database ledger. I seriously doubt if the investment in Blockchain is really worth for enterprise computing that usually takes places between known & trusted partners. Ledger Database that I have mentioned here has been implemented successfully in the past by and it is only a design pattern. Database can be designed to act more like a blockchain without the surprises that under construction blockchain technology has to offer. Bitcoin has been successfully running on block with few exceptions-where system was hacked. Databases are not hack proof either but the benefit of creating a new database to work a Ledger Database comes very close to achieving the benefits that Blockchain offers without fully sacrificing the control of your database which is important for enterprise computing. Let's keep discussing , build come prototype for comparison and find a new cost effective design pattern using proven technologies

Whats is wrong with Aadhar Software System ? (Part-1)

You can read this post Whats is wrong with Aadhar Software System ?  (Part-1) on this link  :
https://digitaltechnologyarchitecture.blogspot.in/2018/01/whats-is-wrong-with-aadhar-software.html

 

I can't say I was shocked to hear about The Tribune (link) report on how their reporter purchased Aadhar data for 500 INR and for another 300 INR purchased a software that could print Aadhar Card ( India's Social Security Number card) but I was surprised at the poor design and operating model of the system. I expected the system to be much more smarter, secure and foolproof than it seems to be now. UIDAI has denied the data leak but Tribune seems to have material evidence and they have published a response to UIDAI statement today (link).

This kind of data leak does not happen because of  a single point of failure. There has to be issue with the
1) Solution architecture (architectural description of a specific solution)
2) Application architecture (describes the behavior of applications),
3) Security architecture (unified security design that addresses the necessities & potential risks),
4) Operation architecture (defines control of operational procedure & execution of operational tasks)
5) Finally the Software testing plan seems to have failed to detect the flaws in the software.

As long as there is valuable data, hackers will try to attack the system but fortunately this was not a hack or else hackers would have wiped the data and held Indian government for ransom. Politicians who are enforcing Aadhar just don't seem to understand a badly designed software can play havoc and that delayed software system is better than a faulty software system. I don't think we need any more proof to call for an immediate software audit conducted by 2 independent reputed agencies. If we are so sure that system has no flaw then software audit will help regain the confidence of  billion Indians. Every big software is audited periodically so why not audit Aadhar System which is a critical system for the government as well as billion Indians. I don't understand why government has not initiated a software audit after multiple 'reported' incidents of data lapses over last 2 years. If the audit highlights any issue then the Aadhar software architects are to be blamed and not the government. The other issue could be the change in Vision, possibly Aadhar system is made to do what it was not designed to do when it designed.. A software audit will only validate that you have a well designed, robust and secure software system and its operations are being managed well by the team.

When  there are multiple issues in a software system they can only be addressed by conducting a detailed software audit followed by fixing the issues identified in the audit. ( Do read my earlier post  Why you should be concerned about #Aadhar being made mandatory for citizen of India? )  Having worked as a fire fighter on solving complex issues on many complex software systems for fortune 100 companies I must say I have not encountered such a large software system that seems to have been put in production without due diligence. I have seen projects where  leads & managers have been removed on the spot for much smaller issues than what was reported by The Tribune. Software systems are custom made to solve a particular business challenge and if there is a failure in basic operation of the software then it is because the project leadership was incompetent to handle the project right from the solution design stage to development stage and onto the operations stage.

Here is a simple picture that will give you an idea of what all could have gone wrong with Aadhar System, I am not privy to the Aadhar architecture so the picture highlights the potential issues in the software based on what we know from reports on data leaks. I am concerned that a software system has become a prestige issue and UIDAI is defending reports rather than publishing a whitepaper on health of the Aadhar System so Indians don't worry about the most critical software that is going to affect every Indian. I want to find out , what is wrong with Aadhar system? What needs to be fixed to avoid the known issues in future?

My understanding of Aadhar System

< Click image to zoom >

So what are the key take away from this data leak incident?

1)  Critical software system like Aadhar are used for verification of Aadhar user's identity and when user input is passed to the system, the systems responds as verification success or failure. The system should not allow users to download Aadhar number or details of Aadhar users. This is a grave software design flaw and there is no doubt about it whatever UIDAI might claim.

2) The data access should have been restricted at application service layer, data service layer as well as database level for additional security and I think all 3 layers have design flaws.

2) The system should have different levels of user authorization that limits user access to certain services and this authorization framework seems to be flawed or else missing from the system.

3) Creation of new user is an activity that should be restricted to a limited set of super users & if we go by the news report then it seems that a use who is not administrator has the right to create new users without approval of any superior authority and this is poor design and a big security risk. A new user creation process is typically performed by the administrator and for critical systems like Aadhar, there should be an 'approval workflow for user creation' where clearance is taken from 'access control board'  or at least a super admin before creating new users. A ordinary system user who accesses the system for routine transaction should not be allowed to create new user at his will. This flaw allows the system to be misused as it seems to have happened in this case.

4)  Authorities have responded that system activities are monitored so it is surprising that when users are being created at will without approval no one noticed it, when system was being accessed by unknown new users no one noticed it, when user data was being downloaded no one noticed it and even the network administrator did not notice unusual activity on the network when billion records were being accessed/downloaded.

                                                                         

< Click image to zoom >

5) In an ideal system that has Business Activity Monitoring, rules are defined on what services needs to be monitored, what alert should be sent by email or by SMS in case of some user accessing a set of services and what action should be taken by the team who is responsible for monitoring the software. None of these things seem to be in place or else UIDAI would have nabbed the culprits before the whistle blower initially complained to authorities and long before reporter investigated the story.

As I mentioned earlier this looks like case of multi point failure and if one of the checks had failed still the monitoring system should have automatically notified the authorities about the suspicious activities within minutes if not seconds of the suspicious event. My guess is a good Business Activity Monitoring tool which is must for any critical enterprise system is either not implemented or the implementation is flawed.

Obviously there could be other issues in Aadhar system that we are not aware of. For example scalability and availability issues and performance issues - and I have reason to believe that  the way it is being implemented today Aadhar system is going face major performance issues one day. The 1 lakh plus municipal employee in Mumbai are using Aadhar bio-metrics to sign-in & sign-out at work.  (I have got proof that municipal employees get emails twice a day from Aadhar system when they scan their finger prints). Which means Aadhar system is being accessed for employee attendance 2 lakh times every day by Mumbai municipality alone! There are around 4000 cities in India which makes 4000 x 2 lakh = 800,000,000 hits to Aadhar server everyday by municipal employees alone! I assume if municipality is using Aadhar for attendance then other government employees will also be using Aadhar verification for attendance and a conservative guess would be 1 lakh government employees across 4000+ cities in India ( I am not considering the employees working in smaller town & remote areas). This means there will be 1.6 billion hits to Aadhar server everyday just verification of attendance of government & municipal employees!  At 9 am when all employees reach office  there will be at least 80,000 hits to the server every 3 second (assuming 4000  cities , 10 offices in each city, 2 bio-metrics scanner in each office)  Did UIDAI plan for this? Is Aadhar Architecture built to take this kind of peak load?  Why the heck should we use national  bio-metrics server to verify that employee in each city has reached office! Did we create Aadhar for such stupid mundane tasks?  Which other SMART DIGITAL COUNTRY in the world has implemented this kind of verification system for government employee?  Which software architect gave this idea to goverment to use Aadhar for employee attendance verification and what are other redundant uses that Aadhar is going to be used for?  It is absurdly, insanely, mind boggling crazy to implement Aadhar based attendance! Imagine when Hospitals, Railways, Airlines, Jios  & every other company starts using Aadhar for verification we will need a new Aadhar Hardware City to host the servers required to cater to such large population! (Ok! Ok! I exaggerated it! Wont need a new city but a huge number of computing nodes on a Cloud since it is quite likely that Aadhar system is hosted on a Cloud)

In my next post I will explain the potential missing blocks that could lead to such system failure in a software system (again I am assuming The Tribune report is reliable and they have indeed purchased billion records for 500 INR. Damit!)

My next post will be about Whats needs to be fixed in Aadhar Software System ? (Part-2) & after that 3rd post will be on How blockchain or similar trust framework could have prevented the Aadhar Data leaks in Aadhar System? (part-3)

Tech Tip - Microsoft has a utility to fix issues with installation / uninstall of programs on Windows

Fix problems that block programs from being installed or removed 

Link  https://support.microsoft.com/en-in/help/17588/fix-problems-that-block-programs-from-being-installed-or-removed

Microsoft has released another  Fix-It tools to "diagnose and fix program installing and uninstalling problems automatically". A Fix-It is a small portable program that Windows users can run to resolve and repair issues on their operating system semi-automatically.

The Microsoft Support page, which is also the page where the Fix-It can be downloaded, lists the following issues that are repaired by the application:

• Removes bad registry key on 64 bit operating systems.
  Windows registry keys that control the upgrade (patching) data that become corrupted.
• Resolves problems that prevent new programs from being installed.
• Resolves problems that prevent programs from being completely uninstalled and blocking new installations and updates.
• Use this troubleshooter for an uninstall only if the program fails to uninstall using the windows add/remove programs feature.