18) Telcos, challenge of Big Data & even Bigger Challenge of Telecom Fraudsters

Big Data brings an interesting set of technologies like Hadoop, Sentiment Analytics, Predictive Analytics. Big Data also brought a new generation of solutions. The only challenge as I have mentioned before is that technology & business need to sit together to analyze and identify the 'right' use cases before investing in Big Data.
      I have implemented Complex integration solution for telcom companies and realized that telco is a world by itself. An outsider will never realize the complex working of a telco and the huge volume of data that they generate! Telcom companies have always been in forefront of adopting latest technology innovations and rightly so. The ever growing customer base & network of Telco world generates humongous amount of data. Telcos requires their software to crunch data faster, filter out the noise & process faster.  Big Data makes way for Distributed Data Processing hubs for various Telco use cases. I am listing a few use cases that I have discussed with some of the top think tanks and innovators of Telcos.

Telecommunication Companies, Fraud Prevention & Big Data

One of the biggest challenge faced by telecom companies is fraud. Telecom companies loose more money to fraud then one can imagine. The fraudster use innovative technology to commit fraud so to detect and prevent fraud we need to use Predictive Analysis coupled with Big Data Processing on the large data generated by each phone, each switch, each tower, every second.

Take a quick look at some facts -

1.Telecom fraud is estimated  at $40 B globally and it is the single biggest cause of revenue loss for operators, costing them between 3% and 5% of their annual revenue. With rising competition & extremely low average revenue per user (ARPU), detecting fraud and plugging revenue leaks have become extremely important to reduce costs.


2.One study reports that the internal fraud (40.3%), roaming fraud (11.4%), pre-paid (10.8%), subscription (11.6%) and premium (13.1%) are the most important in terms of losses by values. 

3.Fraud connected to prepaid accounts is much easier to commit and harder to combat, since there is very little information on the subscriber, unlike postpaid accounts, where a credit check is usually done. Entry-level fraudulent activities such as subscription and impersonation are very serious since the cost is coming straight from the bottom line in the form of commissions and incentives.

4.The fraud management becomes more and more important as the new methods of access become available such as Cable networks,  Wireless networks, DSL, Satellite, Metropolitan Optical Networks running Ethernet, Broadband Wireless Systems (radio, microwave, or infrared).

5.Although there is an abundance of data generated by mobile devices and systems a large amount of data is not processed in real time.

6.Telco would like to detect critical events and patterns across all its data sources in real time, perform advanced in-memory analysis in real time and take preventive or corrective action in real time to providing better service to its customer and reducing the financial looses

'Google Now' Commands that you should try on your Android Phone : Tried & Tested

As most of us are aware the command 'Ok Google' on an Android phone invokes an interactive voice assistant that can understand a vocabulary of commands. So when you pick up your Android phone and say “OK Google” your phone goes in a listening mode and you can do things without touching the phone. So what do you say to your phone? What can you say to your phone? What all can Google assistant do for you?

I tried exploring the commands and here’s a list of just about everything you can say to Google Now. Another interesting thing, if Google Now doesn’t get your spoken commands right, you can correct it by saying “No, I said...” and trying the phrase again. The text in braces [] can is not part of the command and can be changed to customize the command.

General Commands

• How old is [Donald Trump]?
• Where was [Donald Trump.] born?
• Define [colloquial] (Or “What does [colloquial] mean?”)
• What time is it in [Mumbai]?
• Search for [photography tips]
• Show me pictures of [the Leaning Tower of Pisa]
• Do I need an umbrella today? What’s the weather like? What’s the weather in [New Orleans] [this weekend]?
• What the [Google] stock price? What is [Apple] trading at?
• What’s [182 yards] in [miles]? What is [12 ounces] in [liters]?
• What’s [135] divided by [7.5]? (A great many types of math equations will work.)
• Search [Tumblr] for [cat pictures] (more apps are added to this search-within-apps function all the time)

Device Control Commans

• Open [apple.com]
• Take a picture (“Take a photo” also works)
• Record a video
• Turn [on/off] [Bluetooth, Wi-Fi]
• Turn [on/off] [Flashlight] 

Productivity

• What’s the tip for [123 dollars]?
• Set an alarm for [1:30 am]
• Set a timer for [20 minutes]
• Create a calendar event: [Dinner with Ajay, Saturday at 9pm.]
• Remind me to [buy coffee at 7am] (try locations! Remind me to [buy coffee filters at BigBazar])
• What is my schedule for tomorrow? (also: What does my day look like [Friday]?)
• Where’s my package? (tracking confirmation must be in Gmail)
• Make a note: [update my router firmware] (also try “Note to self:” This works with multiple apps, and you can even email yourself!)
• Find [Anne Besant’s] [phone number] (Works with all info in your contacts - addresses, birthdays, etc.)
• Show me my bills. (or: My bills due this week.)

Communication

• Show me my last messages. (Then follow voice prompts)
• Call [John] (also works with relationships: Call [sister])
• Call [Mom] on speakerphone
• Text [Sister] [great job on that feature yesterday] (also works with relationships: Text [mom] [I’m not going to be able to pick you up from the airport, period, I’m a bad son, period])
• Send email to [Milind P], subject, [hunting], message, [I don’t think you should drink so much when you go hunting, period]
• Post to [Twitter]: [Oh its raining again!]
• What is French for [My name is Donald]?
• [Send a Hangout message] to [Dad].
• Send a [Viber] message to [Barak]: Hang on, I'm going to get more coffee. (works with WhatsApp, Viber, WeChat, Telegram, and NextPlus)

Navigation and Travel

• Where is the nearest [chinese restaurant]?
• Navigate to [Nariman Point, Mumbai] 
• Directions to [Fisherman’s Wharf] by [bike] (also try “Directions to home” or “How do I get home?”)
• Where is [the Louvre Museum]?
• Show me the menu for [Hell's Kitchen]
• Call [Salarjang Museum]
• Show me my flight info
• Where’s my hotel?
• What are some attractions around here?
• How do you say [good night] in [Chinese]?
• What is [50,000 yen] in [dollars]?
• What’s the flight status of [Jet flight [735]?
• Show me restaurants near my hotel -or- Give me directions back to my hotel (this works if your hotel confirmation was sent to your gmail account)

Entertainment Commands

• Play [solitaire] (also try tic-tac-toe)
• Play some music (opens “I’m feeling lucky” radio station in Google Play Music)
• Next Song / Pause Song
• Play [Happy] (songs must be in Google Play Music on your device)
• Watch [The Lego Movie] (movies and TV must be in your Google Play account)
• What’s this song?
• Listen to TV
• What songs does [Pharrell] sing?
• Read [Hunger Games]
• Did the [Giants] win today? What’s the score in the [Warriors] game?
• What movies are playing [tonight]? Where is [Toy Story] playing?

Sports Commands

• Say a team name to get the latest score during the season.
• When is the next [Warriors] game?
• Where are the [Giants] in the [MLB] standings?
• Who does [LeBron James] play for?
• Who won [the Superbowl]?
• When is the [Stanley Cup final]?

Must try Fun Commands...

Many of these deliver funny voice responses, but normal search results. Turn up your sound!

• What sounds does a [tiger] make?
• Flip a coin
• Roll dice (rolls a single six-sided die)
• What is the loneliest number?
• Do a barrel roll!
• Askew / Tilt
• Go go Gadget [Spotify]
• When am I?
• Make me a sandwich
• Who’s on first?
• Who are you?
• Beam me up, Scotty!

17) Smarter Border Security using Big Data

I have worked on designing Big Data solutions for various industries. My area of particular interest has been Border Security Solutions. There is immense scope to improve the border security solutions by use of various traditional & Big Data sources & make the security solutions smarter, more robust & predictive.

Any country's border can be penetrated by land, air & water through legal and illegal means. So a border security solution needs to consider more non-traditional data sources apart from the existing data sources.

1) Collection of critical surveillance data using sensor technology - Today various types of sensors, videos camera and connectivity solutions have become far more affordable and the data can be collected remotely, transmitted to a central server and processed in a speedy manner to get insights into various security factors that need to be analyzed for security of borders.

2) Telecom Data- Call Data Records,  Tower Data & User Data from telecommunication providers can be monitored and analyzed for identifying and tracking 'potential security threats'.

3) Voice & Data Analysis - Analysis of voice & data in the border areas can provide information about potential threats and breaches in the border area and most of the borders area already use this data for intercepting threat
communication.

4) Social Media Data- Analysis of data from social-media, blogs and internet call services can help in identifying discussions and profiles that could be involved in objectionable activities related to border areas.

5) Transport Data - Knowledge of vehicle owners & tracking of vehicles in the border areas can provide additional data that can be related to various security breaches and identify any new vehicle in the area. Will this information a mechanism can be defined to sanitize vehicle moving in and across borders.

6) People Data - Biometrics data of the residents of border areas should be used to identify 'new people' who have moved in the area. People movement along with various other threat data can be used to identify security threats and also help predict potential security breaches in border area

7) GPS data is collected by border security agencies and used for various objectives. A big Data solution can help analyze the GPS data to potential security threats as well as to help position security personal on the border thus enabling optimum use of security infrastructure

) Finally Big Data Analytics of the said data sources should help build a comprehensive analytics dashboard that can be used by border agency as well as the security personal that protect  the borders using hand held devices.  I would like to discuss Big Data Analytics and Mobile devices in more detail because this is one of the fastest growing areas of Big Data & Mobile.

Master Technology Architect Program 2015 : I received a lapel pin from our CTO Paul Daugherty

It was quite a pleasant surprise to receive Master Technology Architect lapel pin from Accenture  CTO Paul Daugherty. Now I can flaunt my certification on my coat!

What are the SOA Design Patterns every architect should know?

When I meet architects and developers at different forums I see some people consider SOA to be a independent optional discipline. The fact is tears back there was a time when concept of SOA was new and people had to decide if they should do changes to their application to make them SOA compliant. Today we are convinced about the benefits of SOA and it is defacto design principle. I would go to the extent of stating that if you are not following SOA Principles when designing and coding applications i any technology, any programming language then you are doing something very wrong.

My past employer Accenture was a pioneer in SOA implementation and innovation for SOA. I got opportunity to lead the SOA Practice, work with industry leaders and implement multiple projects as a SOA Architect for some of the top clients. The best way to learn is to implement solutions for the most demanding fortune 500 clients and I was fortunate to get the opportunity as the lead SOA architect for Accenture. One of my innovations SOA Service Bench won 1st prize in Accenture wide innovation contest and I was honored by Accenture CTO Don Rippert which probably is the most rewarding thing an architect wants in his career.

This post is about some basic SOA Design Patterns that you should be familiar with. You can evaluate your existing application to see if they implement these patterns and if not then analyze the benefits of  implementing the SOA Design patterns to make the applications SOA complaint. I assure you the benefits are immense in terms of maintainability, simplicity, loose coupling, reuse and atomic code.   If you require any guidance feel free to write to me. I usually respond to queries on weekend.

1. Agnostic Services
Service capabilities derived from specific concerns may not be useful to multiple service consumers, thereby reducing the re-usability potential of the agnostic services. Agnostic services implement logic that is common to multiple business problems. Separating agnostic logic into discrete services facilitates service reuse and composability.

2. Service Declaration
Agnostic services should explicitly declare that they are agnostic. This makes it clear to future designers and builders which services are designed to be reused.

3. Atomic Service Transaction
Services can be wrapped in atomic transactions with a rollback feature that reverses all actions and changes. Transaction management services can be implemented in the component layer and reused by multiple services.

4. Enterprise Service Bus (ESB)
An ESB acts as a message broker between consumers and services. The ESB can perform message transformations, routing and connect to applications via a variety of communication protocols. YOu can use open source ESB products.

5. Service Façade
The service façade sits between a service and a contract. It eliminates the tight coupling between the service and its contract. This is intended to minimize changes to the service if the contract changes. A service can have multiple service façades to support multiple contracts.

6. Long Running Services & Service Callback
A service requires its consumers to call it asynchronously. If the consumer needs a response it provides a callback address. When the service reaches some milestone in processing it messages the consumer with a response. This approach frees resources and is useful when services are expected to be long running.

7. Backward Compatibility & Multiple Service Contracts
A service may support multiple contracts concurrently. This can be done to support backward compatibility (so that when a service changes all the consumers do not have to be updated). It is also done to provide different views to the service for different purposes (thus facilitating reuse).


8. Authentication Broker
In an enterprise scenario an authentication broker assumes responsibility for authenticating consumers. Consumers are issued a token they can use to access services.

9. Message Origin Authentication
Digital certificates are used to authenticate enterprise clients.


10. Message Filtering
Messages are filtered for harmful data before processing.

16) What implementation should an enterprise prioritize - Mobile Applications,SaaS or Big Data?

Mobile Revolution, SaaS & Big Data followed in succession. Each of these technologies changed the playing field and churned the markets. Yet their are over 70% enterprises that have not implemented mobile platforms, over 90% enterprises that have not leveraged SaaS and over 99% enterprises that have not taken any Big Data initiative as yet. Don't worry about the accuracy of numbers because this is my conservative estimate and real numbers could vary by a huge 20% but the proportion of implementation should be very close.

And for the enterprises that have yet to implement any of these 3 technologies it is quite a clallenge, There are enterprises that were on the verge of building their mobile platforms and suddenly got hit by the SaaS wave and then the Big Data wave.
                            Mobile Platform roll out is a challenge by itself and I have worked on projects that have gone back and forth on mobility platform at times because of lack of understanding of the mobility architecture and at other times due to their rigid service layer which would have to be rebuilt to cater to the mobile platform because of the 'Different Nature & Requirements of Mobile User Interface'. The challenge is around using your existing architecture ad capabilities to support the Mobile Application instead of replicating and rebuilding applications for mobile & we can say the challenge is to implement a classic Service Oriented Architecture to reuse the existing services of the enterprise applications.
                            SaaS or Software as a Service is  value proposition and the need of the hour but it will take some time for enterprises to move to SaaS.  To implement SaaS in true sense, apart from the challenges faced by Mobile implementation there are additional challenges across architecture layers. The good part is that three are multiple 'Multi-Tenancy Models' and an enterprise can implement SaaS incrementally and still derive the benefits of the shared model. My opinion is enterprise should start Mobile & SaaS implementation in parallel. Of course the need an experienced architect to guide them and the 1st step of each implementation would be to build a working prototype.It is also very important to educate the employee of the structural and cultural changes that come along adapting these 2 technologies but that is something we will not discuss here.
                          Finally Big Data, the Big Daddy everybody wants to tell the world that they have Got It Tamed but nobody knows whats will be the end result. No kidding! Ask Apple, Amazon, Bestbusy, IBM, Telstra, AT&T and many other who got started with a humble goal. This is because unless you have a clear understanding of the varacity of Bog Data for your enterprise and its applicability to various areas of business it does not make sense to invest in Big Data. So first things first is to study the changing industry landscape, revisit the enterprise vision and update it and then define the Big Data Vision. I think it is clear that Big Data implementation can follow as you start implementing Mobility and SaaS or even better have all three programs run in parallel if you can. I will be happy to exchange learning's with these technologies if you drop me a mail. Cheers!

15) Tribute to my Idol Scientist & Ex President APJ Abdul Kalam (aka. Father of Indian Nuclear Program & Missile Man)

Most hard working people retire at age of 75 and enjoy a relaxed lifestyle. Those who are fit like to travel across the globe visiting countries & enjoying life and there are few 84 years old people like Indian Ex-President APJ Abdul Kalam who continue to work 24/7, sharing their vision & experience with the youth and guiding them. It is interesting that at the age of 84 Mr Kalam would share his personal email-id at a press conferences & invites people to write to him and guarantees a reply with 24 hours. I have been proud of the fact that I always reply to my business emails, the same day I receive them but I cannot imagine myself sharing my email-id with entire India and replying to any question on any subject 24/7 like Kalam did, it takes Mr.Kalam to Just Do It.
                   I become Mr.Kalam's fan when I heard his interview after 'Pokhran' nuclear test which made India became a nuclear power under his leadership. Most people don't realize the impact of becoming a Nuclear power and if you read last few years of history you will realize the change in world's outlook towards India pre/post becoming a Nuclear power. Interestingly it took two most honest and non-violent people Atal Bihari Vajpaye & Dr Kalam to finally make it happen. When a reporter asked him 'Why does India need a nuclear bomb?'  Dr.Kalam's reply was ' Today the world's top 4 nations are nuclear power and if India wants to be 'Heard as a leader' we cannot sit and do Tapas ( Tapas or Tapsya meaning Doing Penance)'. To be heard as a leader India needs to be one of the top 5 powerful nations and that is why India had to develop its nuclear program.'  If you have heard his lectures would know that Dr. Kalam approach was always Practical, Precise and to the Point and he conveyed his ideas in a very polite way. I like his examples of what a good leader should be like. He said a good leader should face the world and own teams failures & when the team succeeds the leader should let the team take the center stage and he himself should take a back seat - I wish we had more and more leaders following his example. There are far too many leaders who are too eager to own the credit that at times rightfully belongs to someone else or the entire team. He said leaders are watched by people and followed so their integrity should be beyond question and at 84th year of his life he enjoys the popularity because of his honesty, integrity & character. Kalam said a leader should be courageous and should be able to take difficult decisions. I have seen leaders who have been scared of taking tough decisions even when the risk was minimal and the leader fails to understand that a wrong decision put his team back by ages. I wish Indian political leaders would start following Kalam's principles if only to become popular and succeed like Dr.Kalam because no Indian has tasted success like Dr.Kalam in recent years.

I can think of few leadership principles that may not be part of Dr Kalam's speeches but I have observed these qualities in Mr.Kalam and many other good leaders.

1) A leader should be friendly and approachable to everybody in his team - remember Mr.Kalam himself used to read emails sent by anybody & ever body even after he retired.

2) A leader should regularly communicate with his team. Leader's communication should be crisp and to the point and he should listen to the spoken and unspoken.

3) Leader should seek opportunity to appreciate his team rather than finding faults in his team and the team will work hard to seek his appreciation


I did not have the fortune to meet the great APJ but I always felt I had a connect with the man. I guess each patriotic Indian today feels that he has a connect with APJ Abdul Kalam and that is what makes him People's President - a loving title that no other president in the world has been bestowed with by the people. I am happy my son has a modern Indian Idol to follow and I can tell him stories of the great Dr APJ Abdul Kalam.

Rest in peace Sir, Dr Avul Pakir Jainulabdeen Abdul Kalam. You will always be loved and remembered by India

Session Token In URL Vulnerability

The HTTP protocol and web servers are stateless by nature. This means that there is no way for them to track user activity. The web server treats every request as a new one. For this reason, browsers and web servers need to use session tokens. Session tokens are unique pieces of information shared between the browser and the server. They make it possible to track user activity and differentiate between users. For example, an e-commerce application may use a session token to identify the shopping cart that belongs to a particular user.
There are different ways to share session tokens. They are most commonly included in cookies but alternative methods are quite widespread as well. Such methods include sending the session tokens directly in URLs, in dynamically rewritten URLs, or hidden in the HTML source of the web page. These methods are also often combined and on the rise because users often disable cookies in web browsers due to privacy concerns.
Why Is Using Session Tokens in URLs a Bad Idea?
The easiest method of sharing session tokens is placing one directly in the URL, for example, http://www.example.com/account.php?token=12345. Using such an URL, a user who was authenticated earlier can access their account. This method is not inherently insecure but if the session token is not validated by the server, it could lead to potentially high-risk vulnerabilities.
If you place a session token directly in the URL, it increases the risk of an attacker capturing and exploiting it. Anyone who follows that URL inherits the session. When you connect to the web server using HTTPS the risk is less than if you use HTTP but it is still a threat.
HTTPS URLs are encrypted during transmission but they are often stored in server logs. Anyone who gains access to the logs can exploit these tokens. In the worst case, this can lead to session fixation or session hijacking. Therefore, even though we classify the Session Token in URL vulnerability as low severity, you should not take it lightly.
What Are the Alternatives?
Applications should use alternative methods of sharing session tokens, for example, HTTP cookies. You should also encrypt such applications because it is possible to retrieve session tokens from unencrypted applications.If you cannot use cookies, you can send session tokens using hidden input fields. Unfortunately, this is also not foolproof. Attackers may explore the HTML source code to identify and hack hidden fields used to send these tokens. For more on Session hacking read my post about

What is Session Hacking?

14) Enterprise Vision Needs To Be Revisited Before You Prioritize Your Big Data Journey

Big Data planning has to support the enterprise vision and that is why it is critical to revisit and update your 'Enterprise Business & Technology Vision & Road-map' and then start planning for Big Data. This will ensure that  Big Data solution will help the enterprise
1) To identify the Big Data sources within the enterprise and outside the enterprise
2) To identify the Big Data sources that can be tapped for information
3) To filter the Big Data sources that have lower information value
4) To prioritize the Big Data adaption
5) To ensure Enterprise targets 'low hanging' fruits and can demonstrate value from Big Data sooner

An agile enterprise that continues to update its vision and road-map frequently will ensure that the investment in Big Data helps achieve its goals faster and with lower investments.

13) How Big Data Can Fuel Bigger Growth for Your Company

Why should you care about Big Data?

In today’s challenging environment, how will companies grow? The answer is the most valuable yet untapped asset - Customer Information.  Consider how companies are using 'sensors & telematics systems'  in cars to offers lower insurance premiums to its car customers, How online retailers are integrating social media to understand the customer sentiments, how credit card companies are Customer Behavior data to target direct marketers.  It is clear that companies are opening new organic revenue streams by tapping their data & building an information services growth engine.

What can companies do about it?

Before launching a new information business, companies should assess whether they have enough information to differentiate themselves in the marketplace, have data elements that are difficult to replicate in the marketplace, and can combine their data with information from others sources & use sophisticated analysis to create differentiated products. If the assessment reveals there is sufficient revenue potential from an information services business, companies should consider :

1. Consider the best applications of your data (Ex- Marketing, Risk Management, R&D, Operational effectiveness)
2. Compare the attractiveness of the information service to the service offered by existing players (also validate the information services strategy with potential strategic customers) 
3. Understand which important capabilities or assets they will need to win business in the information services 
4. Assess the legal, privacy and policy implications of monetizing information assets
5. Design the most appropriate go-to market approach
6. Use an accelerated prototyping approach to get the new offers into the market quickly

How do you get the started on the journey?

Focus on following 7 step could help companies get started on using Big Data to build a information service business

1. To what extent does your organization use customer data to drive growth?
2. Have you launched new products, services, or businesses focused on the value of your customer information? If so, how successful have those offers been in the marketplace?
3. Do you feel your customer information-based offers are sufficiently differentiated?
4. Do you have a detailed information management strategy tailored to your organization’s unique requirements?
5. How does your organization prioritize information sources as they relate to corporate strategy?
6. With the proper analysis of data from multiple sources, could you develop more innovative and competitive offerings?
7. How much potential insight do you think is locked up in inaccessible data sources?

12) Recap - What is Big Data?

Quick & short recap on 'What is Big Data?'

Today we have got computers is in our pocket, in cash registers, in cars, in TV , in our credit card & everywhere else. Data is being generated by these computers (aka devices) and it is up to the companies to decide if they want to tap this data.  The hardware cost have gone down considerably and companies have realized that they can afford to store more of their data & this analyzing this data will give them a 'more detailed insight of what they are, what their customers want, how companies can improve themselves, how to retain existing customer & get new customer & also save operations cost at the same time'.

According to a research estimate Health Care industry can save up to 300 Billion $. These are just numbers but the fact is Heath Care industry is an early starter and companies have used Big Data to optimize their processes and deliver better, faster,. There are transport companies that are using big dat to optimize the trains running on the lines by optimizing the routes, by optimizing their engines and are helping companies save billions of dollars in fuel cost.

The change has started and companies are adopting Big Data based on the opportunities. The fact is with Big Data related technologies it is possible to store large amount of data, process it in 1/10th of the time and derive value from the data. Companies who adapt to the change over next 10 years will survive the competition. Just collecting data is not enough and unless the data is analyzed companies will not get the value from their Big Data. 

11) A Big Data Use Case - Retail Industry

Let's take an example of a Online Retail Company to understand how they can leverage today's software technologies to leverage Big Data

Challenge faces by online retail company:

•The need to examine massive amounts of unstructured social media and search data to find out what are the 'products that consumers are talking about'

•The growing data volumes causing a major storage problem - leading to data regret on a regular basis

• The need to strategize the ad buying strategy on sites like Google, with the goal of competing for e-commerce sales

• The need to track products, sales, and customers (pet bytes of data) to win pricing concessions from suppliers.

Solution

• Primary basis of solution is co-location of storage and compute layer

• Solution proposes using 'Hadoop' for efficient data transformation

• Solution proposes large proportion of analysis to be performed by Hadoop, MapReduce

   Result

* Over-night processing of data now completes in minutes each day, enabling faster and improved search results

* Data volumes are reduced by as large margin of more than 60%

* Faster analytics that quickly react to changing customer sentiments & market conditions.

Technologies in the solution

• Hadoop, MapReduce, Hive , Pig, Flume, Pentaho, Java 

10) Value of Big Data - All that is old is not Gold & all that is Big Data is not Valuable

The Raw Data from various channels like sensors, instruments, social media usually has to be processed to filter the 'noise or junk data that has no business value' before it be consumed by business system. 

When Big Data is created by a system (a device or an enterprise application or an external source like Facebook) it can either be directly consumed (Ex- For Complex Event Processing) or the data has to be pre-processed before storing in database and this moving data is called as 'Data in Motion'. 

When data is finally stored in some database or a warehouse it is called 'Data at Rest'. There are different benefits that can be extracted from 'Data in Motion & Data at Rest' and the chart below explains the typical steps followed in Big Data processing before Big Data is leveraged for some business outcome. To perform analytic on the data it has to be pre-processed and stored in a database - analytics cannot be performed on raw Big Data.

9) How are Digital Technologies changing the way enterprises (& governments) work?

Digital technologies are changing the way businesses works and also the way employees do their work. I will try to explain the value of Digital Technologies and how they help build Real Time Enterprise Systems that help companies & government to continuously improve & improvise their business process to have a competitive edge in business. In the image below I have explained high level steps to build 'Real Time Enterprise Systems' that leverage standard enterprise data as well as Big Data ( In the image below read, outer circle 1st & then the inner circles)

1) Traditional & Big Data Sources : For any enterprise data is being created by
sensor devices, instruments, emails, social media and the data is of text,audio,video and audio format. Lots of the data is 'noise' and data needs to be cleansed and filtered using smart algorithm before it can be consumed as 'information'
2) Integrated Data & Systems :  Enterprise has many systems & applications and some of them are in silo. A integration bus is essential to integrates the systems and insures smart & secured flow of information across the enterprise.

3) Processed Data : Enterprise runs by taking informed decisions by applying business rules & Business processes to the business data. Data processing & data enrichment is an essential part of a smart enterprise that can take 'Real Time' decisions. Data Processing also involves removing 'noise' from the Big Data that is being created by various sources.

4) Integrated Business Process  : BPM (Business Process Management), Rules Engines, Portal, Mobile applications are the technologies that  help implement integrated BPM in the enterprise and help build an smart enterprise that  takes informed decisions in Real Time as the business events occur in any department in any part of the world. Another advantage of BPM is that a large amount of business decisions can be automated (& work 24/7) & thus business processing can be accelerated. End to end integration ensures that any 'notable event' across the enterprise is monitored & automated decisions are taken in real time.

5) Real Time Enterprise taking Intelligent Business Decisions: The most competitive enterprise have to be Real Time Enterprises. A real time enterprise is an smart enterprise that can take decisions in real time based on information made available to the business. A real time enterprise reduces manual work and uses software to automate a large number of business process and reduces human intervention. The circle in the center of the image above is 'Intelligent Business Decisions' which is the value of implementing a smart real time enterprise by implementing the 4 step steps in the outer circles of the image above. I hope this quick overview was helpful. Do message me if you have any questions.

8) Making a success of Digital India Program kicked off by PM on 1st July 2015

Digital India Program is a new project by Indian government to extend e-governance to the "Gram Panchayat' thus connecting the central government to the basic governance body in smallest entity of our population which is 'The Village" (Gram is Hindi word for Village").

Is Digital India a Revolutionary Step? Yes, it is. This is the first time theIndian  government has committed to connect 'basic governance body of the country to the central governance system' and allocated funds for the program. It is a big thing. This kick starts the Digital India in reality because 'Villages or the Gram' form 75% of India and unless they are connected 'In Real Time' the government is not connected to its people.  The program, according to Govt of India has already got 4.5 Lakh Crore by companies which is very promising.

Here are some key steps that government should take to ensure that Digital India is a success

1) Connectivity : Internet connectivity for all villages in India - Major Challenge
2) Training: Computer Education for office holders of villages & districts - Minor challenge
3) Infrastructure: Electricity or generators for each 'Gram or Village' to use computer.
4) Data Storage: Data is the most important aspect of e-governance. Governments will have to plan for PPP to build a Cloud Data Storage that can be used by Public & Government bodies - from Gram Panchayat and above.
5) Security - Biggest concerns of e-enabling any data is the security risk.
6) Application Design: Mobile friendly design for all government portal applications - Minor Challenge
7) Scalable & High Availability Applications: Existing systems will not be able to handle the huge load of new entities and the applications have to be made scalable and I see this as a major architecture challenge.
8) Continuous Improvement: No systems is perfect and there is need to setup 2 way Communication between Government & Gram Panchayat to get feedback and improve the systems and make them more user friendly and robust
9) Banking:  Banks have to make mobile banking more user friendly and they should reach villages. Public sector banks will have to really push their M-Banking for Digital India to be successful

These are the top 9 concerns that the government should address for successful Digital India initiative.