Data Visualization : Jump Start Plotting With R

Why is data visualization important?

Data Visualization is important because of the way the human brain processes information, using charts or graphs to visualize large amounts of complex data is easier than poring over spreadsheets or reports. Data visualization is a quick, easy way to convey concepts in a universal manner – and you can experiment with different scenarios by making slight adjustments.

Preparing your organization for data visualization technology requires that you first:

• Understand the data you’re trying to visualize, including its size and cardinality (the uniqueness of data values in a column).
• Determine what you’re trying to visualize and what kind of information you want to communicate.
• Know your audience and understand how it processes visual information.
• Use a visual that conveys the information in the best and simplest form for your audience.

 Here are some simple ways to plot data using R tools.

 

1. Basic scatterplot - Main arguments x and y are vectors indicating 

   the x and y coordinates of the data points (in this case, 10).

   Code-

   plot(x = 1:10,
        y = 1:10,
        xlab = "My-XAxis",
        ylab = "My=YAxis",
        main = "Graph Title")

2. Using Transparent Colors in plots - Example of basic plotting with color using "yarrr" package transparent color - Most plotting functions have a color argument

    (usually col) that allows you to specify the color of whatever your plotting.

   Code-

   plot(x = pirates$height, 
        y = pirates$weight, 
        col = yarrr::transparent("blue", trans.val = .9), 
        pch = 16, 
        main = "col = yarrr::transparent('blue', .9)") 

3. Using default R Colors in plots - Most plotting functions have a color argument

    (usually col) that allows you to specify the color of whatever your plotting.

   Code

   plot(x = pirates$height, 
        y = pirates$weight, 
        col = "blue", 
        pch = 16, 
        main = "col ='blue'")

4. Plotting scatterplot with arguments - Example of plotting with arguments. he plot() function makes a scatterplot from two vectors x 
   and y, where the x vector indicates the x (horizontal) values of the 
   points, and the y vector indicates the y (vertical) values. 

   Code

   plot(x = 1:10,                         # x-coordinates
        y = 1:10,                         # y-coordinates
        type = "p",                       # Just draw points (no lines)
        main = "My First Plot",
        xlab = "This is the x-axis label",
        ylab = "This is the y-axis label",
        xlim = c(0, 11),                  # Min and max values for x-axis
        ylim = c(0, 11),                  # Min and max values for y-axis
        col = "blue",                     # Color of the points
        pch = 16,                         # Type of symbol (16 means Filled circle)
        cex = 1)                           # Size of the symbols 

5. Histograms are the most common way to plot a vector of numeric data.

   Code -

   hist(x = ChickWeight$weight,
        main = "Chicken Weights",
        xlab = "Weight",
        xlim = c(0, 500)) 

6. Barplot typically shows summary statistics for different groups. 

   The primary argument to a barplot is height: a vector of numeric values which will generate the height of each bar.

   To add names below the bars, use the names.arg argument.  

   Code

   
   barplot(height = 1:5,  # A vector of heights
           names.arg = c("G1", "G2", "G3", "G4", "G5"), # A vector of names
           main = "Example Barplot", 
           xlab = "Group", 
           ylab = "Height")

    
7. Pirateplot is a plot contained in the "yarrr package" written specifically by, and for R pirates The pirateplot is an easy-to-use function that, unlike barplots and boxplots, can easily show raw data, descriptive statistics, and inferential statistics in one plot.

     Code -

     yarrr::pirateplot(formula = weight ~ Time, # dv is weight, iv is Diet
                   data = ChickWeight,
                   main = "Pirateplot of chicken weights",
                   xlab = "Diet",
                   ylab = "Weight")

Finally, we can save these graphs as pdf file using pdf function of R

Code pdf(file = "D:\MyPlot.pdf",   # The directory you want to save the file in
    width = 4, # The width of the plot in inches
    height = 4) # The height of the plot in inches

# Step 2: Create the plot with R code
plot(x = 1:10,
     y = 1:10)
abline(v = 0) # Additional low-level plotting commands
text(x = 0, y = 1, labels = "Random text")

# Step 3: Run dev.off() to create the file!
dev.off()

Why Nano GPS Chip embedded Indian currency note cannot be a reality

The TV channels, news papers and social media has been talking about new Indian currency of 500 & 2000 INR denominations that were supposed to have Nano GPS chips! Great idea that could make life easier for the government but an impractical one. More ever it that would be illegal and breach of privacy law for a government to track currency notes real time using GPS technology.
              Anyway we are only interested in the technology part so lets discuss the technical feasibility of the idea. If it were not illegal for the government to trace currency in your pockets, cupboards, briefcases and other secret hiding places then it would be every governments dream project to implement so called 'nano GPS based currency notes' and ensure each note leaving the treasury is accounted for, that there are no fake notes in circulation and all the notes traveling across the world to exotic locations and the safe tax havens could be tracked by the government revenue agencies sitting in a room.  GPS technology requires a server that can position a device and a device that can use internet to send its location to the GPS server which means the 'Currency notes with GPS chips' would have to send signal to GPS server and for that they would require a power source and that is what makes the idea unimplementable, even if the countries law allowed tracking of currency.

Why would a government want to use GPS Chip in Currency notes if it were not against privacy laws?

1. Every illegal activity involves paper money transaction to avoid leaving traces
2. Illegal activities like drug trade, fake currency, terrorism, bribery are huge industries that are affecting economies of all countries
3. Tracking of genuine/fake currency notes is not possible at point of services and so it is not easy to control the fake currency notes that get pumped into the economy
4. Monitoring the high taxpayers (& non taxpayers) and their currency transactions would be possible if notes could be tracked
5. Government can (and does) track all your electronic transactions but not the currency transactions without ability to track currency notes

As the Finance Minister said in future he would like more and more people to use electronic money to curb corruption and ensure all people pay taxes that are due to them. 

Digital for Health Care : CBC Monitor application for Cancer Patients

We have been discussing Digital India, Digital World and yet one of the most important segment of our life, Health Care still lacks Digital initiatives. I have made a simple android application to help cancer patients keep track of their CBC and BCR-ABL records and get rid of the paper reports. Do share the CBC Monitor app with people who may find this application useful. The next version of the CBC Monitor will have more advanced features where you would be able to get reports converted to Graphical format by importing them from your mail or Google drive.

https://play.google.com/store/apps/details?id=com.smartmob.projects.activity 

Description from Google Play :
About CBC Monitor App
CBC Monitor is an app to maintain your CBC record (complete blood count) in your android mobile or tablet. The application is useful for people who have to keep track of their Complete Blood Count.

What is CBC or complete blood count?
A complete blood count (CBC) is a blood test used to evaluate your overall health and detect a wide range of disorders, including anemia, infection and leukemia (cancer).

How can CBC Monitor helps you?
1) You can Save all your CBC reports in digital format & get rid of the paper reports
2) You can view your CBC report in graphical format
3) You can compare each parameter of your CBC report across past reports
4) You can share the reports with your doctor, your hospital, your insurance company & your family directly from your mobile.
5) CML (Chronic myeloid leukemia) and Caner patient who have to do CBC test every month will find this application helpful to save their records, view then as a graph and share them with their doctor using phone, email, Whatsapp, Viber or any other communication software installed on their phone.

What mobile devices can I install CBC Monitor on?

Currently the CBC Monitor application can be installed on Android mobile and tablets from Android 4.0 (ICE_CREAM_SANDWICH) onward. If you are a cancer patient and are using an older version of android you can mail us and we can help create a custom application to get it working on your older Android phone - for no additional charges. It can also be used for non-cancer patient who want to maintain their CBC reports.

Support Email : smartmobileideas@gmail.com

Begining Your Data Science Journey

There are tons of data science resources but we often get confused on which resources to follow. I am sharing some steps I followed to learn data science on my own as a beginner. You can also check the links at the end of the article your learning and getting hands-on experience in Data Science.

Programming Language is must to start with Data Science

Whether you are a programmer or new to programming the first step while starting the Data Science Journey is to know programming language. Python is the most preferred coding language and is adopted by most Data Scientists. It is easy to understand, versatile, and Python supports various in-built libraries such as Numpy, Pandas, MatplotLib, Seaborn, Scipy, and many more. The 2nd preferred language for data science is R. Both Python and R learning resources are freely available on internet.

Learning SQL is important when you are working with data

Most programmers are expert on SQL and have worked with 1 or more databases. Structured Query Language (SQL) is used for extracting and communicating with large databases. When you are working with tons of data it is important to know how SQL is used to store & query data. You should have a good understanding of normalization, writing nested queries, group-by, performing join operations, etc., on the data and extract in raw format. This data is then processed using Python , R or any other library.

Cleaning Data is an important step of data processing

When a Data Scientist starts work on a project he has to deal with raw data which is not clean and can't be used for meaningful operations. One has to learn which libraries to use for cleaning the data set, removing unwanted values, formatting data to required format, handling missing values and purging unwanted data. It can be achieved by using some inbuilt python libraries like Pandas and Numpy.
When the data volume is small we can use MS Excel to process the data but Excel has limitations of volume and NOSQL and RDBMS database are used for storing volume data.

 Data Analysis is performed on cleansed data

Exploratory data analysis is the essential part when talking about data science. The data scientist has many tasks, including finding data patterns, analyzing data, finding the appropriate trends in the data and obtaining valuable insights, etc., from them with the help of various graphical and statistical methods, including:

A) Data Analysis using Pandas and Numpy
B) Data Manipulation
C) Data Visualization

You can learn basics of Exploratory Data Analysis from this blog posted by Prasad Patil  

What is Exploratory Data Analysis?

Learning Machine Learning 

According to Google, “Machine learning is a method of data analysis that automates analytical model building. It is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention.”

Here is the list of commonly used machine learning algorithms. These algorithms can be applied to almost any data problem:

1. Linear Regression
2. Logistic Regression
3. Decision Tree
4. SVM
5. Naive Bayes
6. kNN
7. K-Means
8. Random Forest
9. Dimensionality Reduction Algorithms
10. Gradient Boosting algorithms

Some Useful Links

• FreeCodeCamp Website : https://www.freecodecamp.org/
• Kaggle Website : https://www.kaggle.com/learn/pandas
• AnalyticsVidhya Website : https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/
• Deep Learning Website: https://www.deeplearning.ai/
• Google Digital garage : https://learndigital.withgoogle.com/digitalgarage/course/what-is-data-science
•  
  

 

Planning For A Smart City

Smart city services are born from a desire to be a good steward of public funds in the same way that innovations of the past were.Municipalities have always sought to provide top-quality services to their constituents as efficiently as possible and smart city initiative takes the service delivery to next level.

Planning smart city requires both a well articulated planning at the city level and a involvement & support from the community at large. Using data insights to streamline city services requires changing the physical services delivery and also the way administration works to approaches city planning. Creating smart city is not possible without involvement of the people, elected representatives,administration and private service providers. I was invited by one of the officials of the Smart City team to look at their new office and meet his team. It was an interesting experience and I got an opportunity to understand how the local administration was planning to integrate various services under the smart city initiative. On the path to becoming a smart city, the first implmentations are the most crucial to your project’s overall success. Being a software architect I could not help but give the team an overview of how we help Fortune 500 client to implement their project from defining the vision to a successful implementation. Like any new implementation , smart city planning is a set of processes to ensure that you are engaging your residents when necessary while also effectively managing the elements of your plan that are within your control. Every city has its unique footprint and one needs to understand that though the strategy remains the same the roadmap for Smart City implementation will be unique to each city.

1. Prioritize your city's priorities

Your local city community needs will drive which technologies and data you decide to collect first. As a city government, your priority may be anything from improving sanitation or stay animal management to improving transportation within city limits. You may even have existing services in place but may need a better system for managing existing assets. Way to start would be to conduct and audit, review your existing processes and identify areas that are in need of improvisation and innovation and then create a preliminary wish list to inform your planning process.

Once you have conducted an audit, it is important to engage your community to determine what common pain points that are affecting the people. This can be done at a public forum or town hall meeting, using any existing feedback mechanism or conducting contest to determine the most innovative ideas that your citizens can come up with. Local insights are invaluable as you decide where to most effectively allocate your resources, so get creative with how you solicit feedback. The services that affect most people if prioritized will deliver better support for next stage of implementation.

2. Define the vision for your City as Smart City

Very few people know that for implementing great software companies invest a lot in creating great training programs for the employee who are going to use the software. To get your community on board with your vision and start winning champions to your cause, you need to ensure that everybody is working toward a consistent set of goals. Smart cities are meant to be population-centric, so your goals should be measured against the impact your services will have upon the daily lives of your constituents. You may even consider making mission statement to guide your initial project and to provide a touch point to encourage future innovations. As an example, Columbus, Ohio “has a bold vision to be a community that provides beauty, prosperity and health for all of its citizens.” With this mission in mind, Columbus outlined a series of smart systems that led them to win the U.S. Department of Transportation Smart Cities Challenge in 2016.

3. Identify the business model

Many cities have a grand vision of what their smart city could look like, but not all communities can fully implement smart technology on their own due to budgetary or personnel constraints. You may be able to build, own and operate your own system, or a full public-private partnership could be attractive. Evaluate all of your options, including any models that exist between the two, to determine an implementation model that makes the most sense for your city.

4. Perform a gap analysis

A gap analysis is a method of assessing the differences in performance between a business' information systems or software applications to determine whether business requirements are being met and if not, what steps should be taken to ensure they are met successfully.  In order to evaluate your existing infrastructure and identify the steps necessary to realize your Smart City vision, conduct a gap analysis. If you are unsure of how to get started, there are several templates and tools available online that can be found with a quick search. Focus on determining what types of data need to be collected, and be sure to identify the technologies you would like to use relatively early in your process. This way, you can identify the seams and overlaps between different systems, reducing the likelihood of incompatibility issues arising later in your implementation.

5. Outline financing and budgets

While your budget will inform your implementation model as noted above, at this stage it is imperative to focus on short-term, mid-term and long-term implementation ranges. Now is also the time to build a business case for any efficiencies that you expect to gain through the implementation of smart technology. Even this far along in the process, it is a good idea to sell the benefits of your vision to stake holders.

6. Capture the low-hanging fruit

Low hanging fruits give quick and assured results. With a clear view of your budget, identify and group existing assets that are readily scalable to city-wide use. For example, you may be able to integrate your existing transportation infrastructure with utilities and community services. This is the time to focus on the big picture, and you will want to have access to any data and visuals that will allow you to do that. Look to your geographic information system (GIS) or technology department to perform scenario analysis using digital mapping or similar platforms to understand potential areas where connectivity would be required and to identify weaknesses. Parcel data, zoning, and land use information available through location intelligence will all be vital when the time comes to scale your deployment following the pilot phase.

7. Develop and implement pilot projects

When the time comes to deploy your new smart systems, be as targeted as possible with how you roll out your pilot program. Start small in order to better maximize learning opportunities and measure your early successes, and look for an early win you can use to create momentum and positive buzz in the community. Regularly review your smart city vision against the available data to ensure your plan will grow with your community’s needs. If you are struggling to replicate early successes as you expand your offering, then look for patterns in the data that may illuminate ways that you can do so.

Ultimately, the long-term success or failure of any large-scale implementation of smart technology will be determined by the measurement and optimization efforts that take place after deployment. However, by mindfully applying these few steps, you can be assured that your community has placed its best foot forward on the path to becoming a smart city.

Every Child Should Get Education & Nutritious Meal

I have been supporting Akshaya Patra for many years since I came to know about this great idea called 'Mid Day Meal'. Many children from poor families do not get a nutritious meal and Akshay Patra along with government of India provides meals to such schools ensuring no kid remains hungry. The food provided to school is especially prepared for the children at kitchen created & maintained by Akshaya Patra which ensures the quality & nutritious value of food meets quality standards. The food is transported from kitchen to schools in special vehicles and served to children in most hygienic way. I came to know that as of Aug 2016 Akshaya Patra is feeding Mid Day Meals to more than 1.5 million children daily which makes Akshaya Patra world’s largest (not-for-profit run) mid-day meal programme serving wholesome food to over 1.5 million #children!

Being associated with the NGO and having visited their kitchen and schools gives me the satisfaction and confidence that at least 1.5 million students are getting nutritious food and quality education in government schools and the number will keep growing as more and more people will donate to this cause. We all want to do something for the society, we are all concerned that our hard earned money should be utilized in most efficient manner and we want to see the proof that the money is utilized well. I think everyone should visit the Akshaya Patra kitchen to see their modern kitchen where they prepare food for the kids and that will convince you to sponsor Mid Day Meal of at least one child by donating INR 750 (USD 11) or or 3 children INR 2250 (USD 33). When you visit the kitchen or schools you will hear motivating stories of children and some sad stories on how some parents could not afford a balanced meal for their children. I have heard stories of parents who did not send their children to school but now they send the children so that the child can get a meal and education at the government school. Fact is government sponsors such initiatives from the tax they collect from you and me and they have limitation on how many schools they can support from the tax money. People like you and me who want to do their bit and do not have time to contribute can sponsor meals of 3 children every year ( at approximately 2200 Rs which is cost of one dinner for 2 people at a 3* hotel) we will ensure that 3 children will not drop out of school because of poverty and hunger.

Do visit the web site of and know more about their. And don't forget to do your bit every year! Cheers!

Digital India is also about using technology to maintain pothole free roads

Last week I was on vacation and traveled to the beautiful & historical city of Aurangabad. What could have been a wonderful vacation turned out to be a bone jarring experience while driving to various historical spots in the city.  The locals told me that the excuse given by local government is they are not aware of the potholes and as soon as they are informed they try their best to fill the pot holes. The local government should be concerned about the potholes because apart from being danger to life of residents (In India more than 10000 people die because of accidents due to pothole), potholes are bad news for tourism as well. Every tourist friendly country across the world boasts of good roads and neglecting the roads will have adverse affect on the tourism revenue.
             In Digital India governments cannot have such flimsy excuses of not doing their job of maintaining the roads. The local government should be using technology to report, locate and monitor occurrence of potholes. Almost every civic official carries a smart phone and the accelerometer in a smart phone can sense when a vehicle encounters a pothole or sudden dip in the road. With some effort we can build an algorithm that can use accelerometer data, GPS data to build an application that can be used to report potholes and even the severity or size of the potholes. Using crowd sourcing large amount of data can be collected which can be further processed to validate the pothole locations. Smartphone used by drivers of City buses and government vehicles that travel across the city could be primary source of pothole data.
                                I anyone has any experience on building such an application do share your feedback with me. I hope mayors of Aurangabad and other cities that face problem of potholes are made aware that with minimum investment and some incentive to their employees government can build a system that will ensure their road are monitored 24/7, potholes reporting is automated and they can even cross check the claims of contractors who are supposed to fix the potholes saving millions for the government and residents.