How can we add more data to OpenStreetMaps easily?


File:OSM Logo.svg

When exploring on creating maps in Tamil using OpenSteetMaps found that the current data available on OSM is not equivalent with Google Maps.

We can not copy data from Google Maps and import to OSM as it is a big copyright violation. Yes. We dont have control over the data for the area where we live.

We can do the following.

  1. Look for the data sources who may have the data about the streets, villages, cities. I think think the government departments like Postal, Revenue,Rural development, Public Works may have these data. How can we ask them to share the data in public? Will a RTI help on this? Do you have contacts with the leads of these dept? Please help to get map data from them.
  2. Add the data manually. Edit OSM just like how we are adding content to wikipedia. Drawing roads, marking important places is easy. Watch the below video for a demo.

    By this way, we can add any data manually, edit, improve the existing data. But editing on the browser is not possible for many.

    It will be nice if we have a mobile app to add data to OSM. When I had a smartphone, few years ago, searched for a OSM mobile app. Cant find any app, which helped to edit OSM.

    I dont use smartphone nowadays.

    If you have a smartphoe, can you search for the apps, which can help to edit the OSM easily?

    The app should be very simple. The user should should open the app. It should capture the latitude, longitude from its GPS or mobile tower. Then it should ask for the name of the building,  building number, type of the place, street name, area name,city name, if required a photo, etc. Once the user entered these data, it should be synced on OSM. Contribution should very simple as filling few forms only.

    If we have such app, we can create communities/volunteers to add data to OSM with their smartphones. Just open app, fill data. They are done.

If there is no such easy editing app for OSM, it is high time to create such one. If you are a mobile developer, please create such app and help to build the Openstreetmaps as content rich.

Wondering how the Google, Apple, Bing maps collected data. What kind of mobile app they used, what ate the data they collected. If you have worked for these maps, please share more details about them. It will help a lot.

There may be other easier, better ways to add data to OSM. Please share the details or connect with the communities.

Let us build content rich Open Street Maps.

 

Image source : https://commons.wikimedia.org/wiki/File:OSM_Logo.svg  – CC-BY-SA

 

 

 

 

 

 

 

Why do I love apache Kafka?


apache kafka க்கான பட முடிவு

From Wikipedia,

https://en.wikipedia.org/wiki/Apache_Kafka

<quote>

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a “massively scalable pub/sub message queue architected as a distributed transaction log,” making it highly valuable for enterprise infrastructures to process streaming data. Additionally, Kafka connects to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.

</quote>

I am using Kafka as a message queue, in one of my projects that get huge amount of real time data. I get around 6000 to 1,00,000 events per minute. I tried to read those events by a custom python script. The script can not read that huge data. It missed many data.

 

Was looking for a stable data reading tool. Found Kafka and explore it. For my surprise, it worked well. Stress tested with the tool “siege“, producing millions of test data. Single Kafka server received all the data and  stored.

apache kafka க்கான பட முடிவு

It compresses all the data as its own internal format and keeps them all. By default, it stores for a week. Anyone can write to it and anyone can read from it, in a very stable process.

Logstash is a perfect pet for reading from kafka. Then it can write to s3, another kafka or elasticsearch.

Installation is very simple. Just download, extract, start running it.

https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04

With confluent platform, it can read and write json documents easily.

I strongly suggest to use kafka on any message queue requirements.

Image sources:

 

 

Real Time Bigdata Analysis – Few Tools


https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAZbAAAAJGE5Y2ZiNmU2LWRhNTgtNDhlYi05YTY0LTAwYWVmY2EyZGY5Yw.png

 

Big Data Analysis is becoming one of the hot words in the IT industry. Everyone wants to analysis data. They all want to use the tools like hadoop, spark etc. These are used to process huge amount data. i.e in TB size . This is called “Historical Data Analysis”.

In opposite to this, there is “Real Time Data Analysis”. This is to process immediately on the stream of constantly incoming data.

The typical data pipeline for Real Time Big Data Analysis is as below.

App/Site->API Server->Message Queue(Kafka) ->Processor(Logstash)->Storage(Elasticsearch, Redis, MongoDB)->Visualization(Kibana)

Few years ago, we had to rely on Google Analytics and pay huge amount of money to get real time data of our site visitors, credit card swipes etc. Nowadays, we can build entire pipeline with Free/Open Source Software itself.

https://i1.wp.com/blog.infochimps.com/wp-content/uploads/2012/05/realtime-analytics.png

With the following links, we can setup the data pipeline easily.

https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04

http://docs.confluent.io/3.2.0/kafka-rest/docs/index.html

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

To setup these things are easy. But once the real time flow is started on production, remember, you are always on fire. You will feel that you are riding an aeroplane, with so many buttons on the dashboard. You have to keep running, while solving the real time issues when they appear.

Explore these tools and learn their basics. Learning Basics will give their sweet results for sure.

There are tons of new tools coming in this arena. We can not master all the tools. But, exploring and learning one tool will help to keep on moving with new tools easily.

I am exploring the following tools along with the ELK.

  1. Presto
  2. Spark
  3. Secor
  4. Druid
  5. Hadoop
  6. Hive

Doing most of the programming with Python. It becomes very slow to deal with GBs of data. Go language seems faster to work with text files. Started exploring Go too.

What are the new tools, technologies you are learning?

 

Image source- https://www.linkedin.com/pulse/real-time-stream-processing-big-data-platform-birendra-kumar-sahu

http://gcastd.com/

 

 

Looking for a social media manager for FreeTamilEbooks.com


We are looking for a social media manager for FreeTamilEbooks.com

It is a voluntary task.

The roles are as below

  1. Publish the new ebook arrivals at mailing lists, facebook, google plus, whatsup, telegram groups
  2. Update the XML file https://github.com/kishorek/Free-Tamil-Ebooks/blob/master/booksdb.xml for new ebooks
  3. Monitor Social media and response to queries from readers
  4. Contact bloggers and writers to get new contents

If you are interested in volunteering for this, send an email to FreeTamilEbooksTeam@gmail.com

Join in our forum
https://groups.google.com/forum/#!forum/freetamilebooksforum

and post your interest.

 

Filed a RTI to get info on Tamil TTS by IITM and SSN


I blogged on the topic “How to ask IITM to release IndicTTS as Free/Open Source Software?” recently.

Had a good discussion about this on the ILUGC mailing list too.
https://www.freelists.org/post/ilugc/How-to-ask-IITM-to-release-IndicTTS-as-FreeOpen-Source-Software

As a followup on this,  filed a RTI on this.

We can request for any information online in the portal https://rtionline.gov.in itself. Registration asks for our address and phone number. Then we can fill the request form.

Note : The content box does not allow Question mark and URL.

Asked for the below information.

1.
Is there any government policy or G.O to release the software developed by or funded by Department of Electronics and Information Technology (Deity) and Ministry of Communication and Information Technology (M CIT) as Free/Open Source software

2. Is IITMadras funding SSN engg college to develop a Tamil Text to speech software

3. If so, how much is the funded amount

4. Send me the project plan, roadmap, and cost splitups for the development

5. There is a open source android app for Tamil text to speech at IITM site. (IITM donlab site)
This is very very old. But the latest development by SSN college available at  [ speech DOT ssn DOT edu DOT in ]  is very new and works well. Why it is not released as Free/Open Source Software with source code

6. When can we get the latest Tamil Text to speech software from SSN college, as free/open source software with source code

Once submitted, paid Rs 10 via its online payment gateway. It was smooth.

Received an acknowledgment as below.

https://i0.wp.com/storage9.static.itmages.com/i/17/0310/h_1489149948_9704514_0b02428daa.jpeg

Fine. Let me wait for 30 days for the responses.

Will share the results here.

 

 

 

Is it possible to create a system for anonymous RTI?


rti க்கான பட முடிவு

Last week, I read a book about Right to Information, Thagaval Ariyum Urimai Chattam (Tamil) Kindle Edition

Thagaval Ariyum Urimai Chattam  (Tamil) by [S.A.M.Barakath Ali]

This book describes the history, usages, processes to apply for RTI, and about the murders, threats for the RTI activists.

The history, usage, examples are interesting. But, it seems that we may get murder threats, or get murdered by the unknown people. Few RTI activists lost their life because of asking questions about the scams and brought them out to public.

Sometimes, to bring bad name for any activist, fake RTI applications are produced by their names. For example, to give trouble to an activist, fake RTI was raised asking about a temple on a road side. Govt people tried to demolish the temple. When people asked, they told his name and he only raised RTI. The anger people yelled at him. Then he  had to explain that he did not raised that RTI.

All these happens, because, we have to give our Name and full address when applying for RTI.

Currently, there is no option to apply as anonymous. When we give a track to find us, they will find us for sure.

Thinking on a system to make RTI applying fully anonymous. This is a mixture of software, hardware and good hearted, bold people.

Currently, we can apply for RTI ourself, or via any organization like DYFI. Still, they need our address. If they put their address, when someone asks for the source of the RTI, they may give our details.

What we need is, the organization should not know any details about the RTI applicant.

File:Anonymous – CeBIT 2016 01.jpg

 

Imagining the following system.

  1. There should an organization, which can collect requests from common people, via online form or email or mobile app, anonymously.
  2. They apply for RTI with their address
  3. They announce the RTI application info with a unique id on their site
  4. Once the Information is received, they publish the results with the unique id on their website
  5. The original RTI applicant, checks the site and with the unique id, he can find the info.

To hide all traces of the RTI application, he should use Tor network systems. If the information from the govt is too private, we should explore the system of Public Key, Private key encryption systems to transfer the information between various networks.

There is a online site that does RTI request on behalf us. https://onlinerti.com This asks for our address and online payment of Rs 200. The proposed organization can act similarly.

The next problem is online payment for the organization. Is there any way to send money anonymously online? if not, the organization should get donations and make it as a free service.

If we can create such an organization where we can hide our details 100%, more people will come forward to apply RTI for public issues.

Is it possible to create such an organization?

How can we use the latest technologies to provide 100% anonimisity for the RTI applicants?

Share your thoughts.

 

Image credits  :

https://www.newslaundry.com/2016/09/29/indias-rti-act-is-the-4th-best-in-the-world

http://bit.ly/2m38037

 

 

 

Getting started with LXC containers


We, sometime need more than one GNU/Linux machines to test some network services, firewall rules, port checkings etc. I was using VirtualBox to install another GNU/Linux os. Used it for all the networking needs.

Using VirtualBox is really heavy. It virtualizes entire os and separate base OS and guest OS. With my 8GB ram laptop, I can run hardly 2 virtual box instances. Was looking for a simple, light weight virtual machines. Found Docker. But it seems to learn a lot. I need even simple one, just to roll out 10 or 15 virtual machines, in my laptop itself.

lxc container க்கான பட முடிவு

LXC Containers:

Found LXC containers. This is very very lightweight. I am running 20 containers in my laptop and created a good network to try various activities. It shares most of the basic resources with the base OS itself. Separates only the minimally required components.

The beauty is I can run many versions of Ubuntu in the same machine.

Few commands to use LXC:

Install:
sudo apt-get install lxc

 

Create a container:
syntax : lxc-create -t <template> -n <container name>

Example :
lxc-create -t ubuntu -n ubuntu1604

 

To get a older version of ubuntu,
sudo lxc-create -t download -n ubuntu1204 — –dist ubuntu –release precise –arch amd64

To list all the containers:
lxc-list

To get information about a container:
sudo lxc-info -n ubuntu1604

To start a container:
lxc-start -n <container name>
sudo lxc-start -n ubuntu1604

To get inside the container:
sudo lxc-attach -n ubuntu1604

 

There are more interesting commands like lxc-clone, lxc-monitor, lxc-snapshot, etc. Exploring them.

Thanks for LXC developers. It makes my life easy. Found that Docker uses LXC as its base. Will explore docker too.

Image source : http://blogs.cisco.com/enterprise/what-the-heck-is-a-service-container

Few links to explore about LXC:

http://blog.scottlowe.org/2013/11/25/a-brief-introduction-to-linux-containers-with-lxc/

https://linuxcontainers.org/lxc/getting-started/

 

How to ask IITM to release IndicTTS as Free/Open Source Software?


https://i2.wp.com/purvispublicsafety.com/wp-content/uploads/2014/12/text-to-speech.jpg

Indic TTS

This is a project on developing text-to-speech (TTS) synthesis systems for Indian languages, improving quality of synthesis, as well as small foot print TTS integrated with disability aids and various other applications. This is a consortium based project funded by the Department of Electronics and Information Technology (Deity), Ministry of Communication and Information Technology (M CIT), Government of India involving 13 institutions and SMT, IITM being one of them. The project comprises of Phase I and Phase II. Phase I of the project used Festival-based speech synthesis for Bengali, Hindi, Tamil, Telugu, Malayalam and Marathi . Phase II of the project commenced in 2012 employing HTS based statistical speech synthesis for 13 Indian languages.

Source : https://www.iitm.ac.in/donlab/tts/

As a part of IndicTTS system, SSN college people have developed a good working TTS for Tamil. You can see the demo here – http://speech.ssn.edu.in/

IITM has released a very priliminary version of Tamil TTS as android app as open source here – https://www.iitm.ac.in/donlab/tts/androidapp.php
That too without proper license and attributions to base, open source HMM HTS systems.

But the web version is really awesome and with great human like voices.

As the entire project is funded by Government of India, I think it should be released as Free/Open Source Software.

I requested the IITM team at https://www.iitm.ac.in/donlab/tts/contact.php to explain about the status of releasing the latest version of Tamil TTS as Free/Open Source software.

Did not get any reply.

We have the two options now.

  1. Improve the basic version of oepn source Tamil TTS they have released. But, need to explore the provided license to improve further and release in GPL or similar license. For this we have to do huge research on TTS and spend years again. Why do we have to reinvent the wheel?
  2. Ask to release the current TTS as Free/Open Source software. This seems a better as there is already a working software. Need to know whom to ask and how to ask.

Like to know that if there is any govt law says that all the software funded by public money should be released as Free/Open Source software. If so, please share the details about it.

Next is whom and how to ask for it.

Do we need to file a RTI? or need to request via any legal firm? is there any legal organization to support these kind of activities? can we get support from political party like communist to move it further?

It may be tougher to get the source as it involves more money, power, fame, policies, politics etc. But, we should ask for our rights as the TTS is made by our own money.

Share your thoughts.

 

Image source – http://purvispublicsafety.com/2014/12/06/text-to-speech-dispatch-centers/

Celebrate 42 lakhs downloads of Free Tamil Ebooks with Us


4milliondownloads

On July 26, 2013, we started the FreeTamilEbooks.com Project, to create and share Tamil Ebooks in Creative Commons license. We release the ebooks in 4 formats, ePub, Mobi, A4 PDF and 6 inch PDF, so that we can read the books on any device like kindle, boko, nook, android, iOS, Desktop, Laptops etc.

Read more about he project here – http://freetamilebooks.com/about-us/

We added the download counters for each book. But wanted a full report on downloads count. Waited for three years. My wife, Nithya was learning Python programming. Gave this as assignment to get the report of all the books download count. She wrote a script, which crawls all the book pages, get the contents, search for the download counts, add in a report.

Here is the report –
http://freetamilebooks.com/htmlbooks/download-report.html

Here is the source code-
https://github.com/nithyadurai87/fte-ebooks-download-counter

With this report, we can see the most downloaded books on file type wise,and total wise.

Good. Found that there is a number for total download count too.

It says as  4228868 now. This is 42,28,868 i.e 42 lakhs, 4.2 million.

Wow. 🙂

Never thought that we will reach so many readers. This means there are good Tamil readers are there around the world, who are ready to read the good contents.

Tons of thanks for all the contributors, writers and readers for making this project as a useful one for the Tamil readers.

kindle-woman-reading

Thanks to the following communities, who are behind this project.

With all the technical and content support from all around the world, the project is growing well.

If you are interested in contributing to the project, join our forum and share your interest.

Join our forum here-
https://groups.google.com/forum/#!forum/freetamilebooksforum

Image source-
https://theeventscalendar.com/celebrate-4-million-downloads-with-us/
http://cbdsports.com/

Let us create Maps in Tamil


Maps. I have seen them in school days only on charts, globe at HM Desk and on Atlas books.

The growth of Maps is very exponential. Nowadays, Maps plays a important role on transportation. Along with GPS, now we can reach most parts of the world without even asking a single question to anyone.

The GPS devices says the right path in English. It has been a long question for me to look for Tamil announcements on GPS. To build such system, we need Maps in Tamil. Then Text to speech system.

How can we build Maps in Tamil or such non-english languages?

My friend Srikanth Lakshmanan < srik.lak AT gmail DOT com > explored on this and found some answers.

See his results here

h_1488649559_2669274_5a8c1fee0f

https://api.mapbox.com/v4/srikanthlogic.714e671e/page.html?access_token=pk.eyJ1Ijoic3Jpa2FudGhsb2dpYyIsImEiOiJuQ1RYS3pjIn0.7YUMcAQAc4A7T703-yAu2g#3/23.08/78.31

He wrote about it here.
http://blog.srik.me/2016/10/04/indic-maps-using-mapzen-tangram-openstreetmap/

He uses Mapzen,Tangram and OpenStreetMaps.

The results are impressive.

explore this repo for https://github.com/srikanthlogic/tangrams-indic
and contribute for further development.

What next?

We need Tamil strings for all the English strings. All the city names, town, street, nagar areas are only in English.

We have to translate all of them in Tamil.

Started a project in https://translate.zanata.org/ for a community contributed translation.

Testing it to incorporate with google translate api so that we can translate the missing words only.

The tests are going here.

https://translate.zanata.org/webtrans/translate?project=TamilMap&iteration=1&localeId=ta&locale=en-US&dswid=-8936#view:doc;doc:tamil-map.txt

Will announce once all the test are completed.

Is there any already a collection of English city, street names and equivalent tamil translation available online ?

or can we ask TamilNadu government to give if it has such a collection?

Can MozillaTN or such communities can lead this to have this map translation as one of their activities? So that we can get more volunteers and take it to other languages too.

Share your thoughts on this.

Dreaming my mobile shows route to a new area in Tamil and GPS says the route in Tamil. Can you see the same dream? Come. Let us make it real.