GSOC 2016/Student Application bee2502

Contact Information

• Your name:Bhagyashree
• FAS Account: bee2502
• Fedora userpage: Bee2502
• Email Address: bhagyashree dot iitg at gmail dot com
• Blog URL: https://networksfordata.wordpress.com
• Freenode IRC Nick: bee2502
• Github : https://github.com/bee2502

Why do you want to work with the Fedora Project?

I love Fedora OS

While Fedora isn't the first Linux distribution I have used, it is surely one which I have used the longest and am most comfortable with.

I love the Fedora Community

The Fedora community is very warm and welcoming. I especially like that CommOps encourages contributors to work in diverse areas and to try out new stuff, with the Fedora community always ready to help you out if stuck.

I love Fedora CommOps

I love the work. I love the team and I want to continue contributing and helping improve Fedora. Period.

High Impact

Even as a newcomer, I have had the opportunity to work on high impact projects like organizing elections or working on metrics which affect strategic decisions. The huge impact your work can have on milllions of Fedora users and contributors is something which motivates me to contribute to Fedora.

Great Learning Opportunity

Due to the flat hierarchy in Fedora, I have already collaborated with or worked under some of the long term contributors and important figures in the Fedora community. This experience has been a great learning opportunity in many different ways and I look forward to many such chances in the future.

I look forward to work and be involved with Fedora. I aim to stick around and become a long term contributor in the Fedora community.

Do you have any past involvement with the Fedora project or any other open source project as a contributor?

Yes, I have been involved with the Community Operations team since the past six months. Some of my past contributions include -

Statistics related Contributions

• Data Analytics to understand impact of FOSDEM : read here and code here
• Year in Review metrics for Fedora CommOps : read the report with information about API queries, analysis and data visualizations here
• Community Blog statistics : read the report with information about API queries, analysis and data visualizations here
• Outreachy Impact metrics : read here and related API query here
• F23 Dec/Jan Election related metrics : read here and related statistics here, here and here
• IRC metrics using fedmsg activity and datagrepper : read here and code here
• Spammer Activity in Fedora - some graphs ML thread here and related API query here and here

Other Technical Contributions

• Contributed to Fedora Hubs for gaining technical knowledge of the codebase which would be helpful in developing metrics related widgets.

Issues I have fixed include :

https://pagure.io/fedora-hubs/issue/106

https://pagure.io/fedora-hubs/issue/96.

You can see my closed PR's here

My Hubs related fedmsg activity here

Other Contributions

• Collaborated with Justin Flory on Women in Computing and Fedora article.

• Helped Jan Kurik organize F23 Elections ! They were the fourth most participated elections in all time.Read more about the F23 elections on the CommOps retrospective here.

• Completed the CommOps Join Process. Wrote a Fedora Badges post to aid Newcomers : 'How to get started with Fedora Badges?' read it here.

• Helping in diversity and women outreach efforts of Fedora by being an active member in Fedora women community.

• I have also contributed to the Fedora Community Blog (see my works here) and to Fedora Magazine(see my works here)

• I have a good knowledge of the wiki,Trac,IRC and mailing-lists and I am comfortable with using them to communicate effectively. I have communicated and interacted with my mentors and other team members in IRC and ML and having been involved with CommOps, I understand the ethics and values that make up the Fedora Community.

• You can see my overall contribution activity via fedmsg here and here

Apart from Fedora, I have done an Open Source Data Analytics project for Measurement Lab

Did you participate with the past GSoC programs, if so which years, which organizations?

No

Will you continue contributing/ supporting the Fedora project after the GSoC 2016 program, if yes, which team(s), are you interested with?

I will, of course. I'll continue with the CommOps team and Hubs development. I am also interested in being an Ambassador(but that's for a bit later)

Why am I the best fit for this project idea?

I am really passionate about Data Analytics. With data, I want to understand and impact the community by bringing to light the critical issues along with identifying our strengths and weaknesses to help the leadership make informed decisions. My proposal for the Community Operations slot for Fedora in GSoC revolves around this idea. Along with sound technical skills required to implement this proposal, I also feel that I have the required non-technical skills ideal for effective open source contributions

Some relevant points include :

• I'm really passionate about open source, love the CommOps and Fedora community and I will continue to contribute to Fedora and CommOps even when the project ends.
• I am comfortable with coding in Python , C++ , R and can write queries in SQL. I also have intermediate knowledge of HTML and CSS.
• I have working knowledge of fedmsg system and datagrepper queries and have done related data analytics projects before link here
• I also know Machine Learning and NLP and I am interested in using these techniques to understand Fedora community better.
• I am learning Data Visualization techniques like d3.js so that I can develop interactive visualizations from data.
• I also have contributed to Fedora Hubs development in the past and have knowledge of the codebase.Issues I have fixed include :

https://pagure.io/fedora-hubs/issue/106

https://pagure.io/fedora-hubs/issue/96.

You can see my closed PR's here. My Hubs related fedmsg activity here. Being familiar with the codebase, I can help in CommOps related tasks for Hubs development

• My contributions to Fedora have not just been limited to technical aspects. To gain a deeper understanding of the Fedora Project, I have tried to contribute in diverse areas including helping Jan Kurik organize F23 elections(which was the fourth most participated elections in Fedora history) , writing a Fedora Badges article to help newcomerslink here, contributing to Community Blog (see my works here) and to Fedora Magazine(see my works here) and helping in diversity and women outreach efforts of Fedora by being an active member in Fedora women community.

• I blog regularly, and I believe this will help me develop interesting and well laid-out documentation as well as data analytics reports for the project.
• I have a good knowledge of the wiki,Trac,IRC and mailing-lists and I am comfortable with using them to communicate effectively.
• I have communicated and interacted with my mentors and other team members in IRC and ML and having been involved with CommOps, I understand the ethics and values that make up the Fedora Community.

• If you haven't guessed it by now, I really love CommOps and contributing to Fedora and GSoC offers me a great opportunity to do so over the summer !!! Additionally, I get to work on statistics and Machine Learning - what more could I ask ?!

Project Proposal

Overview

Fedora Community Operations(CommOps) : Statistical Simulation and Data Analytics for Fedora Infrastructure Message Bus Activity

Community Operations, a.k.a. CommOps, aims to address the area of community infrastructure by providing the tools, resources, and utilities for the different subgroups of Fedora to increase communication across the Project.

Because of the fedmsg stack, Fedora has very detailed raw data on Fedora contributor activity. My proposal revolves around programmatically querying Datagrepper API for data collection to build automated tools using Statistical and Machine Learning techniques for data analysis and visualization for different parameters.

GOALS

STATISTICAL TOOL FOR FEDORA EVENT ANALYTICS

• Develop automated tools for data collection using tahrir API in conjunction with Fedora Infrastructure Message Bus activity
• Develop statistical tools and algorithms for real-time data analysis using numpy and pandas python libraries.
• Also, identify and code suitable Clustering algorithms for demographical analysis using scipy and sci-kit learn python libraries
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js

STATISTICAL TOOL FOR FEDORA INFRASTRUCTURE MESSAGE BUS ACTIVITY ANALYTICS

• Develop automated tools for data collection using Datagrepper API queries for Fedora Infrastructure Message Bus activity.
• Develop statistical tools and algorithms for real-time data analysis using numpy and pandas python libraries for generating sub-project wise metrics, contributor wise metrics like mean Contributor Age(Fedora Activity wise), Retention Rate of Contributors
• Generate programmatic python scripts for Time Series Modelling of data using scikit-learn and scipy libraries in python.
• Also, identify and implement suitable Machine Learning algorithms(like Temporal Clustering) to find similarity patterns in sub-project activity, build contributions etc
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js

• Develop an automated tool for the fedmsg statistics for quarterly report for Fedora Project.
• Develop statistical tools to identify long-tail patterns in contribution activity(How many contributors are just long-tail from packaging one thing?)
• Use Machine Learning algorithms like Logistic Regression,SVM or Neural Networks to distinguish Redhat vs Non-redhat contributers on lists and conduct a statistical analysis.
• Develop a Temporal Clustering based tool to identify similarity in contribution patterns for long-time contributors ( Do successful/old contributors have diverse contributions ? Are their contributions in bursts or continous over a period of time ? ) (optional)
• Alternatively, statistics tools could also be implemented as statscache plugins instead of automated python scripts, depending on feasibility.

• ML Discussion here , hereand here and related ticket on CommOps Trac instance here and here

STATISTICAL TOOL FOR MAILMAN/HYPERKITTY ACTIVITY ANALYTICS

• Develop automated tools for data collection using HyperKitty API in conjunction with Fedora Infrastructure Message Bus activity
• Develop statistical tools and algorithms for real-time data analysis using numpy and pandas python libraries to generate statistics like mean/median size of ML thread , number of people in a thread, mean length of discussions, redhat vs non-redhat activity.
• Generate programmatic python scripts for Time Series Modelling for ML activity data using scikit-learn and scipy libraries in python to identify activity patterns(bursts/highs and lows).
• Also, identify and implement suitable Clustering algorithms to find activity-wise and trend-wise similarity patterns in lists.
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js
• Alternatively, statistics tools could also be implemented as statscache plugins instead of automated python scripts, depending on feasibility.

• ML Discussion here and here. Related tickets on CommOps Trac instance here and here

STATISTICAL TOOL FOR BUGZILLA ANALYTICS

• Develop automated tools for data collection using Bugzilla API in conjunction with Fedora Infrastructure Message Bus activity
• Develop statistical tools and algorithms for real-time data analysis using numpy and pandas python libraries.
• Generate programmatic python scripts for Time Series Modelling for data using scikit-learn and scipy libraries in python to identify activity patterns(bursts/highs and lows/mean Bug turnaround time).
• Also, identify and implement suitable Clustering algorithms to find activity-wise and trend-wise similarity patterns.
• Identify relevant algorithms and develop Machine Learning based tool to identify easy-fix or most relevant bugs (Optional)
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js
• Alternatively, statistics tools could also be implemented as statscache plugins instead of automated python scripts, depending on feasibility.

STRETCH GOALS

STATISTICAL TOOL FOR FEDORA BADGES

• Develop automated tools for data collection using tahrir API in conjunction with Fedora Infrastructure Message Bus activity
• Develop statistical tools and algorithms for real-time data analysis of badge collection activity using numpy and pandas python libraries.
• Also, identify and code suitable Clustering algorithms for demographical analysis using scipy and sci-kit learn python libraries
• Generate programmatic python scripts for Temporal Analysis for data using scikit-learn and scipy libraries in python to identify activity patterns
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js

FEDORA HUBS WIDGETS

• Componentization of CommOps deliverables into Fedora Hubs Widgets.
• Develop metrics and statistics related widgets for Fedora Hubs

Some other cool Ideas bee2502 would like to work on

• Automated NLP-based tool to find the expertise Fedora Contributers

Develop an automated tool using NLP techniques to find the expertise of Fedora contributors using meetbot logs of IRC meetings. We are looking to answer questions like "Who can best solve my doubt?" OR "Who is the most qualified person for this task?" NLP libraries for python like gensim or nltk will be used for tool development.

• Badge Recommendation Engine widget for Hubs

Much along the lines of Stack Overflow Badge Recommendations : "You are 50% of the way to earning the 'Master Editor Badge' " Provide recommendations like "80% of contributors who last collected 'White Rabbit Badge' went on to collect 'Origin Badge' next "

Develop an automated tool with backend using Tahrir API to fetch data. Identify suitable Recommendation Algorithms like Collaborative Filtering and develop the engine using them

This could be especially helpful for newcomers to explore different areas of Fedora Project Some related representations by mizmo : https://fedoraproject.org/wiki/Fedora_RPG_OLD

• Automated Tool to publish IRC meetings word clouds to social media like Twitter

Generate wordclouds from meetbot data using NLP techniques or wordcloud tools/libraries for python. Develop a tool using Twitter API to publish these wordclouds to Fedora handles on social media.

STATISTICAL TOOL FOR GITHUB ANALYTICS

• Develop automated tools for data collection using Github API in conjunction with Fedora Infrastructure Message Bus activity
• Develop statistical tools and algorithms for real-time data analysis using numpy and pandas python libraries.
• Generate programmatic python scripts for Time Series Modelling for data using scikit-learn and scipy libraries in python to identify activity patterns(bursts/highs and lows/mean issue turnaround time).
• Also, identify and implement suitable Clustering algorithms to find activity-wise and trend-wise similarity patterns.
• Identify relevant algorithms and develop Machine Learning based tool to identify easy-fix or most relevant issues (Optional)
• Provide real - time interactive data visualizations using suitable tools from matplotlib or d3.js

Final Deliverables

• Automated statistics tools(scripts in python,preferably) for data collection,analysis and visualizations to be committed to the fedora-stats-tools github repo and/or Community Operations Toolbox whichever appropriate. Alternatively, statistics tools could also be implemented as statscache plugins instead of automated python scripts, depending on feasibility.

• Data files(.csv) to be committed to the fedora-stats-tools github repo

• Analysis Reports and documentation of work to be published to community via Mailing lists, Fedora Planet and/or Community Blog posts whichever appropriate.

• Report back weekly on Community Operations to Mailing Lists, Community Blog, and other channels when appropriate.

Timeline

I would like to start having a look and master the technical stuff that I need to fulfill even before the Community bonding period starts.

Community bonding period (22nd of April - 25th of May)

• Work on improving the technical skills needed for the project (especially data visualizations)
• Understand the bugzilla API
• Discuss and finalize the Machine Learning Algorithms neccesary for FEDORA INFRASTRUCTURE MESSAGE BUS ACTIVITY statistical analysis tool
• Discuss project specifications with mentor.
• Develop an automated tool for the fedmsg statistics for quarterly report for Fedora Project.

Work Period until mid-term evaluations (25th of May – 20th of June)

• Work on STATISTICAL TOOL FOR FEDORA EVENT ANALYTICS
• Work on STATISTICAL TOOL FOR FEDORA INFRASTRUCTURE MESSAGE BUS ACTIVITY ANALYTICS
• Communicate to the team regarding weekly status
• Update personal blog posts to be syndicated on Fedora Planet as per the progress

Period of submitting mid-term evaluations (20th of June - 27th of June)

• Clean Code and test for bugs.
• Fix related bugs and write documentation.
• Code review by mentor.
• Submitting and completing midterm evaluations.
• Update personal blog posts for weekly status, to be syndicated on Fedora Planet

Work Period (27th of June – 15th of August)

• Work on STATISTICAL TOOL FOR MAILMAN/HYPERKITTY ACTIVITY ANALYTICS
• Work on STATISTICAL TOOL FOR BUGZILLA ANALYTICS
• Communicate to the team regarding weekly status
• Update personal blog posts to be syndicated on Fedora Planet as per the progress
• Also attend FLOCK :)

Final Week(15th August - 23rd August)

• Clean Code and test for bugs.
• Fix related bugs and write documentation.
• Code review by mentor.
• Submitting and completing midterm evaluations.
• Update personal blog posts for weekly status, to be syndicated on Fedora Planet
• Wrap up and Complete tasks

Potential Mentors

Remy Decausemaker(decause) , Corey Sheldon(linux modder) and Justin Flory(jflory7)