Project Idea – Telegram bot to translate strings for Open Source Projects


telegram bot க்கான பட முடிவு

In wikimedia hackathon, I saw a demo of using telegram bot to translate strings from translatewiki.net

here are the notes about it.

============

Telegram Translation Bot: https://phabricator.wikimedia.org/T131664 DONE

Translate on translatewiki.net without leaving your Telegram app

Code: https://github.com/amire80/mediawiki-telegram-bot/

mediawiki.org page: https://www.mediawiki.org/wiki/User:Amire80/chat_bot_draft

Phabricator: amire80 * Wikipedia: Amire80 * Twitter: @aharoni

Amir E. Aharoni and Taras Bunyk presenting

Justin Du (MtDu), Taras Bunyk, and help from Brian Wolff, Madhvuvishy, bd808, Niklas Laxström, Jon Robson, and more people!

“Most people don’t speak English”

Translatewiki.net – thousands of messages to translate

can now translate through this simple mobile app instead of needing to load the full site in a browser

selects untranslated strings, in your preferred languages, sends them to you, and you translate, and it submits them to translatewiki

Long messages are automatically skipped to fit a use on mobile.

============

Thinking as we can build a bot to translate the strings for mozilla and openstreetmaps.

Need to get your inputs/thoughts/ideas for this.

translate க்கான பட முடிவு

These links may help to build a telegram bot for translations.

https://github.com/zanata/zanata-python-client

https://translate.zanata.org

https://translate.zanata.org/iteration/view/TamilMap/1/settings?dswid=1182

use this command to get the po file in /tmp/ta.po

zanata po pull –url https://translate.zanata.org/ –project-id TamilMap –project-version 1 –transdir /tmp

We can process the po file using polib

http://polib.readthedocs.io/en/latest/quickstart.html

There are many python libraries to create a telegram bot.

http://telepot.readthedocs.io/en/latest/

https://khashtamov.com/en/how-to-create-a-telegram-bot-using-python/

https://blog.pythonanywhere.com/148/

https://www.codementor.io/garethdwyer/building-a-telegram-bot-using-python-part-1-goi5fncay
With all these tools to create a bot, to process Po files and zanata to host the translations, we can connect them all.
If any one is interested in programming for this, reply here.

Thanks.

Image sources:

http://www.asktrustdee.com/2016/03/my-top-5-telegram-bot.html
https://commons.wikimedia.org/wiki/File:Translate_en-ta.png | CC-By-SA

Advertisements

Project Idea – Teach Tamil to Apertium – Open Source Machine translation tool


Apertium க்கான பட முடிவு

 

Apertium is a Open source Machine Translation system. It supports many languages like english, hindi, urudu etc.

Explore its features and train Tamil for it.

The following links will help to explore further.

https://www.apertium.org

http://wiki.apertium.org/wiki/Apertium_New_Language_Pair_HOWTO

http://wiki.apertium.org/wiki/Dravidian_languages

https://sourceforge.net/p/apertium/mailman/message/23318108/

 

If you are interested in exploring this, comment here. Let us work together and make some move with apertium for Tamil.

Project Ideas – Part 2 – Looking for contributors


Here are few more project ideas.

1. mobile/web app to record voice for wikisoure – Show a word, record it, upload to commons, link back to wiktionary.

2. mobile/web app to record audio books  – FreeTamilEbooks needs audio books too

3. wordpress to android app convertor – Why cant we convert a wordpress site as android app with RSS feeds?

4. epub to apk convertor – Let us publish ebooks as mobile apps too.

5. blog to epub convertor – fix, add images
https://github.com/sathia27/blog2ebook
Add a feature to download images and add them to ebooks.

6. Daily mobi files for tamil newspapers
Crawl newspapers daily, make mobi, send them to kindle in email daily.

7. Send to kindle – feature for FTE
Add Send to kindle feature to FreeTamilEbooks.com site

8. Lime survey – SAAS – alternate to google forms
Explore limesurvey and make it as alternate for google forms.

9. Collect politicians info and release as app, site

How can we collect all politicians details as education, assets etc and publish for public?

http://tshrinivasan.blogspot.in/2015/12/how-to-collect-details-of-TN-politicians.html

10. setup ELK for tamil literature search, build a search engine on top of it

Explore using ElasticSearch and Kibana for Tamil Text analysis.

11. fix android app to record audio for wiktionary –

https://github.com/Atul22/wikiAudio
done at https://meta.wikimedia.org/wiki/WikiConference_India_2016/Chandigarh_Hackathon

12. Analyse tamil tv/radio show audio, find how many english words are used/hour
This paper may help
https://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf
https://mail.python.org/pipermail/chennaipy/2017-March/001429.html
Contact Ganesh for python implementation of this algo

13. gui for voice record/upload – wiktionary

https://github.com/tshrinivasan/voice-recorder-for-tawictionary

This needs a GUI version for windows users

14. gui for csv uploader

https://github.com/tshrinivasan/tools-for-wiki/tree/master/csv-uploader-wiktionary

This needs a GUI for windows users

15. gui for open-tamil font convertor

https://github.com/Ezhil-Language-Foundation/open-tamil

Need a web application or GUI for all features of open-tamil

16. mobile app to teach tamil – pollachi nasan

http://tshrinivasan.blogspot.in/2015/03/blog-post_9.html

17. wiki massuser create

Sometimes, we need to create 100s of users on wikipedia, for any training/event. Currently, only 6 users can be created. Admins can create multiple users, one by one. Automate this process using mechanize and beautifulsoup.

18. OCR4wikisource web version using google vision api

Rewrite https://github.com/tshrinivasan/OCR4wikisource with google vision api and give a web interface.

19. create a command line TTS from the source of a mobile TTS app.

Here is a open source TTS mobile app for tamil.

http://www.iitm.ac.in/donlab/tts/androidapp.php

Register and download the source and apk.
The voice named “Naveen” is good.

There are many c files in the folder
SSNFlitehtsTamil/app/src/main/jni

Can you compile those files and give a binary file as a command line tool?

Explore these code and share your thoughts on how to convert this as a
desktop/command line application so that we can use it in our
computers.

20. Create a GUI app for bulk photo uploader for http://commons.wikimedia.org

https://github.com/tshrinivasan/mediawiki-uploader

Project Idea – Automation script needed to download British Library books


British Library has already digitized many Indian books (including Tamil, Bengali and other languages) and uploaded them in their website.[1]  The books are split in separate pages in .tiff format, so, we need a script to automate the process of transferring them in Internet Archive/Commons as a single pdf/djvu file, so that we can use it in Wikisource.

https://i1.wp.com/eap.bl.uk/images/header_main.jpg
Got this request from my Wikipedia friend Bodhisattwa Mandal
I checked few Tamil Books.
Example :
http://eap.bl.uk/database/overview_item.a4d?catId=164997;r=18467

“Access for research purposes only” is the license for this file.

But, it seems that these books are very old and already in public domain.
We have all the permissions to download them and publish anywhere.
Now, we need a program in python or any language to download all the books, magazines from the sire http://eap.bl.uk and to provide them as individual PDF files or a zip file of images.
Once, if we get the PDF or image files, we can do OCR them using google OCR and get text out of them. Then, we can publish both images and text for further proofreading and fixing to WikiSource sites, using OCR4WikiSource.
if you are interested to contribute for this project, reply with your details in comment or send mail to tshrinivasan@gmail.com
Thanks.

Project Ideas – Part 1 – Looking for contributors


contribute to open source க்கான பட முடிவு

I am listing here few project ideas and requirements. If you are interested in contributing to any open source project, consider these to start with.

I am giving an intro about each of them in this series of blog posts.

Add your comment here if you pick any of the project to do, so that others can join with you.

1. Clean up Epub files.

We create epub files for FreeTamilEbooks.com by using Calibre. It creates epub files with lot of extra span and other tags. We need to remove all the unwanted tags from those epub files.

Create a command line or web application to clean up the given epub files.

If you are writing in python, plan to create a calibre plugin to clean the epub files.

2.  Download reports for Tamil Wikisource Ebooks

http://ta.wikisource is providing ebooks downloads.

In this database, all language wiki source ebook downloads are stored.

http://tools.wmflabs.org/wsexport/logs.sqlite

Create a web application or command line application to get the details of tamil books and create a download
count report for each book.

Create similar report as http://freetamilebooks.com/htmlbooks/download-report.html

 

3. Improve FreeTamilEbooks android app

The android app for FreeTamilEbooks has some bugs.
https://github.com/jskcse4/FreeTamilEBooks/issues

Use the App and read the issues.
Fix them.

 

4. OCR4WikiSource – Create a web application

OCR4WikiSource is a command line application that connects google ocr and wikisource.
It sends the pdf files to google drive, ocr it, gets text, sends to wikisource.

Create a web application to upload any pdf file, send to google via google vision api, get text, send to wikisource.

Links:
Here is the requirement.
https://github.com/tshrinivasan/OCR4wikisource/issues/89

Few links about it.
https://goinggnu.wordpress.com/2015/12/28/announcing-ocr4wikisource/

https://goinggnu.wordpress.com/2015/09/30/automating-google-ocr-with-python/

https://meta.wikimedia.org/wiki/WikiConference_India_2016/Submissions/Introduction_to_OCR4WikiSource

Discussion with wikipedia developers on this.
https://phabricator.wikimedia.org/T120788

Google Vision API
https://cloud.google.com/vision

Explore the links

https://github.com/GoogleCloudPlatform/cloud-vision

http://terrenceryan.com/blog/index.php/working-with-cloud-vision-api-from-php/

https://github.com/thangman22/google-cloud-vision-php

http://blog.aimanbaharum.com/2016/04/21/ocr-with-google-cloud-vision-api/

 

5. FlipBoard like application for Tamil

Flipboard is a web, mobile app which gives latest content on user selected topics. Create such application for providing tamil content from web on various topics. Content contributors should give links for good articles with relevant categories, tags. Users should subscribe to categories and read the latest content.

 

6. Firefox plugin for tamil wikisource proofreading

 

Tamil wikisource is having around 2000 public domain ebooks, OCRed by google OCR. We have to proofread those books manually.
QuickWikiEditor is a Firefox plugin that enables on the page editing of wiki content.
https://addons.mozilla.org/en-US/firefox/addon/quickwikieditor/

Need to extend this plugin, to send the error words and the corrected words to a remote web application. From there, we can get the list of error words, search for them in entire ta.wikisource.org, replace with the corrected words automatically using bots.

Extend the plugin and create a web application to get the words collection from the plugin.

 

7. Fix the Tamil TTS by IITM

IIT Madras and SSN college, released a Text to speech application for Tamil, as an android application. You can get the source at
https://www.iitm.ac.in/donlab/tts/

It is very initial version. Not as good as the latest  web version available at http://speech.ssn.edu.in/

 

Still, we can learn, extend the initial version.

Explore the android app, get the C code out of it, create a command line app or web app as having the c code as backend.

 

8. Web application to add details about ebooks in a xml file, in github.
We release Tamil ebooks at FreeTamilEbooks.com

We store all the details about the books in a XML file.

This file is source for Android and iOS apps for FreeTamilebooks.

Once an ebook is released, we have to update the xml file manually, which is tough for non-tech contributors.

Need a web application to get the ebooks details in a form, then add those details in XML file and commits to the repo automatically.

 

9. Add ebooks automatically in GoodReads.com

We can add the details about the ebooks in FreeTamilebooks.com to GoodReads.com

We have to fill a long form manually.
Need a command line or web application to simplify this process or automate it for adding info about the books in FreeTamilEbooks.com

10. Build a SAAS version of planet kind of RSS aggregation software.

 

Most tech communities need a planet kind of RSS aggregation software. They have to buy a VPS, install planet software and add the RSS feeds.

It will be good, if we build a SAAS version of planet or similar software, so that they can simply sign in, add rss feeds and start using it.

There are more ideas. Written them somewhere on my notebooks. Will collect them and share soon.

All the projects should be released as Free/Open Source software only.

If you are interested in doing any of the things said above, comment here.

email me to tshrinivasan AT gmail DOT com to know more details on any of the project.