CS-GY 6513 - Big Data - Final Project
Jobly is a cutting-edge platform designed to revolutionize the job search process by leveraging BigData and LLM technologies. It offers a suite of comprehensive features, including:
- Advanced Data Analysis: Jobly utilizes large troves of data, analyzed using powerful big data methods with Spark, to identify trends, analyze market dynamics, and provide valuable insights to users.
- Natural Language Processing: The platform incorporates an OpenAI LLM to enable users to communicate with the data in a natural language format, asking questions and receiving relevant information.
- Comparative Analysis: Users can easily compare job postings based on salary, required qualifications, and job responsibilities.
- Skill Aggregation: The platform aggregates data from various job postings to present a consolidated list of the most sought-after skills and qualifications for specific roles.
- Personalized Recommendations: Jobly leverages user profiles, including qualifications, experience, and preferences, to deliver tailored job recommendations that align with individual career aspirations.
Jobly's user-friendly interface, built with Streamlit and LangChain, allows for seamless interaction and customization. Users can select their preferred GPT model and customize hyperparameters to tailor the platform to their specific needs.
- Efficiency: Streamlines the job search process by providing relevant information and personalized recommendations.
- Informed Decision-Making: Empowers users with data-driven insights to make informed career choices.
- Career Advancement: Helps individuals identify and pursue opportunities that align with their skills and goals.
- User-Friendly Experience: Provides a simple and intuitive interface that is easy to navigate.
- Data Analysis: Leverages big data methods with Spark to analyze large datasets.
- Natural Language Processing: Employs an OpenAI LLM for user interaction and data querying.
- User Interface: Built with Streamlit and LangChain for a user-friendly and customizable experience.
Jobly is more than just a job search platform; it is a valuable tool for individuals seeking to advance their careers and achieve their professional aspirations. By combining advanced technologies and comprehensive features, Jobly provides a one-stop solution for job seekers, empowering them to make informed decisions and find fulfilling career opportunities.
We aim to answer 3 questions for users -
- How does this job posting compare to other similar jobs?
- What skills does this job role usually require?
- Based on my profile, what sort of job role fits me best?
Various technologies used in this project include -
- Python
- Spark
- Spark NLP
- YAKE
- Oxylabs API
- Flask
- OpenAI API
- LangChain
To view more details, refer to the README under each section (data-analysis, data-scraping, web-app, outputs).
Matthew So - BigData
Rohit Mohanty - BigData
Shashank datta Bezgam - Full Stack & LangChain
Ke Zhang - Full Stack & LangChain
Jasmine Wu - Jobs Data Web Crawler
