PAIGE: Examining Learning Outcomes and Experiences with Personalized AI-Generated Educational Podcasts

AI-Generated Audio Content for Learning

Context: College students studying textbook chapters across various subjects. This includes students studying assigned chapters for coursework.

Implementation: Textbook chapters are converted into audio podcasts using advanced language and voice models (e.g., Gemini 1.5 Pro, AudioLM). Implementation involves generating generalized or personalized audio content. Personalization incorporates student profile information such as major, age, interests, and learning style to tailor the content. Students are assigned to different conditions, including textbook, generalized podcast, or personalized podcast.

Outcomes: Students found podcasts more enjoyable than textbooks. Personalized podcasts led to higher learning outcomes in some subjects (e.g., Philosophy and Psychology) compared to both the generalized and textbook conditions. Both podcast conditions received significantly higher attractiveness ratings than the textbook condition. Participants found the podcast format more entertaining and appreciated the casual and conversational style of the podcasts. The AI-generated podcasts replicated the engaging experience in real-time and provided suitable supplements to textbooks.

Challenges: TTS voices occasionally sounded unnatural and exhibited minor audio glitches. Personalization was perceived as irrelevant by some students in some subjects (e.g., Government). Some students found the AI-generated content to be initially uncomfortable, and some disliked the AI-generated content. Personalization did not impact attractiveness ratings. The lack of control over the granularity of personalization may have influenced attractiveness ratings. Findings may not be applicable to students from other cultural contexts.

Technical limitation

TTS voices occasionally sounded unnatural and exhibited minor audio glitches.

Proposed Solutions: Improving audio models should be a priority.

Relevance and Effectiveness of Personalization

Personalization was perceived as irrelevant by some students in the Government subject, and it did not always lead to increased enjoyment (Attractiveness ratings).

Proposed Solutions: Further investigation into how and when personalization should be applied, and how to create content that clearly connects to a user's interests and experiences. Exploring the long-term effects of AI-generated podcasts on learning retention, and the influence of various personalization strategies on different demographic groups.

User Acceptance and Ethical Concerns

Some students expressed a dislike for AI-generated content and felt uneasy about the AI pretending to know them.

Proposed Solutions: Acknowledging potential user concerns and considering them in future design decisions. Exploring how to develop and train models that accommodate diverse cultural perspectives.

Scope and Context

The study was limited to a single textbook chapter per participant, involved U.S. college students, and focused on three college-level textbooks.

Proposed Solutions: Future research should explore longer study durations, a broader range of subjects, demographics, and educational levels.