Aastha's Blog

Redefining Limits: Conquering My First Hackathon Challenge

How I Built a Research Paper Recommendation Service Incorporating SMTP Protocol, an Intuitive UI, and a Feedback System!

Aastha's photo
Aastha
·

6 min read

Index

  • Why participate in a Hackathon?

  • Inspiration for the project

  • Mechanism

  • Challenges we ran into...

  • Accomplishments we are proud of

  • What did I learn from the experience?

  • What's next? The conclusion

Why Hackathons?

Participating in a hackathon is like stepping into Willy Wonka's Chocolate Factory of Tech enthusiasts. It's a magical realm where brilliant minds come together, armed with keyboards instead of golden tickets, to create something extraordinary. Just like a recipe for success, hackathons blend equal parts creativity, collaboration, and caffeine-fueled coding sessions. It's a rollercoaster ride with ups and downs, bugs, and breakthroughs. But amidst the chaos, you forge bonds, exchange knowledge, and unlock your inner problem-solving wizard. So, if you're ready to sprinkle some coding magic and embark on an exhilarating adventure, jump aboard the hackathon train. Your next grand tech confection awaits!!

I have taken part in a hackathon conducted by HackSharks on DevPost, and I have been shortlisted as one of the top 15 finalists out of 500+ participants.

I am currently awaiting the last round, and let's see what this round has in store for me :)

Inspiration for the project

My team member and I are B.E. (Bachelor in Engineering) students and wanted to learn about new architecture in Machine Learning. For this, we had to read the latest/relevant research papers in that field, but when we tried to do so, the websites were confusing and cluttered at the same time! It was extremely difficult to surf those websites to procure a relevant research paper, especially for an undergrad.

This problem has a popular name, "Paradox of Choices," or it is rightly said that "Less is More"!

This was when we thought a recommendation engine for apt research papers was a good project idea to help other students like us find such journals.

Mechanism of this project named: JOURNAL-HIVE

-> Would you like to have a look at the project?

Click on "GitHub Repository" to check it out! You can also refer to the ReadMe file with instructions on how to run the project on your local machine.

System Architecture of the Project

  1. The project is built using MERN stack and Python. When a user interacts with our Web UI, it sends the required data like signup time, user-configured interest, and user email to our backend.

  2. The backend handles the user registration and authentication using JWT tokens and MongoDB. When the user configures his/her interests, this emits an event in Node and creates a child process running a Python script.

     if (process.platform === "win32" || process.platform == "win64") py = "python";

 //executing python script by spawning the child process
 const sendToPython = spawn(py, [`${process.cwd()}/backend/Controllers/userInterest.js/api_test.py`, ...keywords, uid])
 sendToPython.stderr.on('data', (data)=>console.log(data.toString()))
 sendToPython.stdout.on('data', (data) => {})
 sendToPython.stdout.on('end', () => { console.log("Request sent to recommender") })

  3. The Python script then uses SPRINGER API to fetch all the articles Springer offers related to the user's interest. This list of articles is then sent to the recommendation system.

  4. The recommendation system uses a TFIDF encoder to encode every article’s abstract and title. Then similar articles and interests are grouped. This is again cross-referenced with the user ratings of past recommendations.

  5. Finally, a score is prepared to serve the most recommended article. This is saved in DB, which is then used by the mail scheduler to send emails at regular intervals by checking the time of the last mail in DB.

Challenges we ran into...

  • Handling event loops in NodeJS became very complicated with the increase of API endpoints, for instance, asynchronous calls.

  • Initially, we faced many restrictions due to the policies of multiple platforms like Yahoo and Gmail. Scheduling emails through SMTP was difficult because blocking and non-blocking code ran simultaneously.

  • Preprocessing fetched data from the SPRINGER API was a challenging task due to nested JSON and text formatting.

  • Using WordNet to search through synonyms of user interest was difficult since we had to run a few installers while spawning the child process.

  • Integrating the feedback system in the front-end, back-end, and database, we had to develop a new scoring system incorporating user ratings.

Accomplishments that we are proud of!

  • We were successful in designing a scalable, easy-to-implement, and efficient system.

  • Email scheduling which runs smoothly was the most complicated part we tackled through persistence.

  • We learned how to work in a team (remotely) and brainstorm together as a team. That's the best part of participating in a Hackathon.

  • How to properly showcase a demo of a working project with its intricacies and all the documentation.

  • Consistently keeping track of the project's parts by committing them on GitHub regularly was an enlightening experience.

    Is that all? No! Hackathon gives you an opportunity to demonstrate your project and enhance your presentation skills likewise.

    Click "Demo Video" to check out the project's demonstration.

  • What did I learn from the experience?

  • Event loops in NodeJS and how to handle them.

  • Child process spawning and detaching it with the parent process to run it asynchronously.

  • Scheduling emails, using blocking code like setInterval() and setTimeout() with non-blocking code used to fetch database.

  • TFIDF encoding of textual data using sci-kit learn and using cosine similarity to find similar text groups in embedding space.

  • Resolving database conflicts, like VERSION ERRORS and overwriting issues by different processes.

What's next for Journal-Hive?

  • Adding OAuth authentication for Gmail API integration.

  • Integrating multiple languages into the same platform.

  • Building a recommendation engine for platforms like Medium.

  • Developing a ranking system for Q&A platforms like Quora and social media platforms like Twitter.

  • Making a YouTube video recommendation by encoding its captions and transcripts.

Conclusion

In conclusion, my first hackathon experience has been an incredible growth, learning, and accomplishment journey. Building the journal recommender engine with its seamless UI, automated mailing service, and powerful backend has taught me the value of perseverance, teamwork, and creative problem-solving. This project has showcased my technical skills and deepened my passion for innovation and entrepreneurship. I am grateful for the opportunity to participate in this hackathon and be surrounded by brilliant minds worldwide. The experience has left an indelible mark on me, fueling my drive to continue exploring and pushing the boundaries of what is possible. I am excited about the future and eagerly anticipate the results of the finals. Regardless of the outcome, this hackathon has been an invaluable experience that has set the stage for many more exciting endeavours.

Hackathons are where your crazy ideas become a reality. You can earn money, a lot of prizes, meet like-minded communities and feel good about your accomplishment like never before! Do participate in them and learn on the GO!

WeMakeDevsMERN StackPythonhackathon