Lecture digest - auto summarization and key points recognition

Yap, Melissa Chia Chean (2024) Lecture digest - auto summarization and key points recognition. Final Year Project, UTAR.

Preview

PDF Download (4Mb) | Preview

Abstract

Addressing the diverse learning preferences of students who benefit from visual, textual, or auditory cues, we developed Lecture Digest - to automatically summarize lecture videos into multiple types of lecture summary. Lecture Digest provides the option of viewing text summaries, navigating to important sections of the video by clicking chapters, and interacting with chat assistance. In text summarization, we use Google's Speech-to-Text technology to convert spoken words in videos into written text before the transcription feeds into GPT-3.5 to extract important points based on semantic understanding. Meanwhile, in video summarization, we use a custom gesture recognition system based on the MediaPipe framework to recognize hand gestures that can signify important key points. The intuition is that speakers would normally perform some distinguishable hand gesture when emphasizing important points. Then, we mark the timestamp of this landmark into the video for the user to easily skip ahead and rewind to these important segments. Lastly, a chat module acts as a virtual tutor, enabling students to engage deeply with the video content. Users can query the module to pull up information related to the video, review frequently asked questions, and utilize interactive tools like flashcards and quizzes to test their understanding. The experimental results on 60 UTAR students from different faculties showed that all of them are satisfied with the functionalities provided by the Lecture Digest application, proving it to be an effective tool in boosting the student’s academic performance.

Actions (login required)