AI-Empowered Analytics on Gesture

This research explores how students apply bodily engagement for computer-supported collaborative learning (CSCL). We have utilized MediaPipe, a machine learning model, to analyze video files and extract moment-by-moment body landmarks. This approach allows us to understand and visualize how students use hand movements and body gestures during collaborative learning activities. 

How do students apply bodily engagement for CSCL?

 

  • Machine learning model: MediaPipe
  • Input: video file
  • Output: moment-by-moment body landmarks


This research compares students' gesture employment quantity in different consensus building discourses: quick, integration-oriented and conflict-oriented consensus building discourses.

This study reports that students apply significantly more hand movements in conflict-oriented consensus building than in quick consensus building. This indicates that more bodily engagement is applied during idea negotiation than in superficial discussions. The visual representation shows moment-by-moment extracted landmarks from video recordings, providing insights into the dynamics of student interactions. The study highlights the importance of bodily engagement in facilitating deeper levels of collaboration and understanding among students.

  • Students applied significantly more hand movement in conflict-oriented consensus building than quick consensus building.
  • More bodily engagement was applied during idea negotiation than superficial discussions. 

Lyu, Q., Chen, W., Su, J., Heng, K.H.J.G., Liu, S. (2023). How Peers Communicate Without Words-An Exploratory Study of Hand Movements in Collaborative Learning Using Computer-Vision-Based Body Recognition Techniques. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science, vol 13916. Springer.​

Lyu, Q., Chen, W., Heng, K. H. J.G., Su, J.& Wang. Y. (Accepted, 2024). Hands-on consensus building: Leveraging deep learning models to unveil hand gestures in consensus building discourses. Cognition and Instruction.