The Not-Too-Late Show with Elmo S1:E2

Jimmy Fallon / Kacey Musgraves

Before bedtime, Elmo takes on the extra special role of talk show host! And for his very first episode, Elmo welcomes guests Jimmy Fallon and Kasey Musgraves, who sings a Sesame Street classic.

John Mulaney / Lil Nas X

Elmo looks inside some awesome goody bags before challenging guest John Mulaney to a nail-biting tricycle race and duetting with Lil Nas X. Ernie leads a baaa-ckstage tour right in the middle of the show, and Cookie Monster gets a special visitor.

Batman / Pentatonix

Batman is in the house to play a game of Beach Ball Ballyhoo with Elmo. In search of the perfect bedtime story, Elmo welcomes acapella group Pentatonix, who performs a mashup of lullabies. Rosita looks for three sheep so she can tell her favorite sleepy time tale.

Jordin Sparks / Nature Nick

Trainer Nature Nick brings some animal friends on-set to help teach Elmo about nocturnal creatures. Jordin Sparks sings one of her favorite childhood songs and shares the thing she loves most about nighttime. Correspondent Grover takes to Sesame Street to investigate whether or not the sun should be in bed. Correspondent Grover takes to Sesame Street to investigate whether or not it's night.

Miles Brown / Joyous String Quartet

Elmo faces off in a nursery rhyme rap battle with Miles Brown, and the Joyous String Ensemble drop by, in their PJs, to play a medley of familiar tunes. Cookie Monster babysits his little niece, who runs loose around the studio before her bedtime.

Blake Lively / Dan + Shay

Elmo and Cookie Monster talk weird dreams before Blake Lively returns to Sesame Street for a game of "Cookie Monster Says." Dan + Shay perform their own rendition of "I Don't Want to Live on the Moon." Oscar the Grouch prepares to finally get his chance in the spotlight for "Trash Talk."

John Oliver / Kwame Alexander / Sofia Carson

Elmo finds out what animal John Oliver would be, and the two compete in the world's slowest and most dramatic race. Sofia Carson joins the slow-mo fun with a lovely lullaby, and Kwame Alexander recites an impromptu poem about - who else - Elmo!

Hoda Kotb / Ben Platt

Elmo does some hard-hitting journalism to learn about Hoda Kotb's morning routine before challenging her to a game of "Hot Potato." Ben Platt treats Cookie to a song about his very favorite thing. Grover offers his expertise on how to pack Elmo's backpack for school.

Mykal-Michelle Harris / Jonathan Van Ness / H.E.R

Elmo brings in "monster makeover specialist" Jonathan Van Ness to unleash the gorgeous monster inside Sesame Streeter Alan, before Elmo and Cookie unveil hair and makeup transformations of their own. Mykal-Michelle Harris gives hosting a whirl and faces off with Elmo in a "No Laugh Challenge," and H.E.R. performs a song about her beautiful, versatile hair.

Andy Cohen / Josh Groban

When Grover gets lost on his way to guest-star on the show, Andy Cohen steps up to the plate and pretends to be Grover. As a talk show host himself, Andy knows the need to keep the show going no matter what and through his Grover impressions, Andy fools everyone into thinking he is, in fact, Grover ! Finally, Andy reveals his true identity while Grover still searches for a way onto the talk show set. Josh Groban performs I love Trash with Oscar the Grouch.

Olivia Wilde / Sara Bareilles

The Count helps Elmo keep an eye on the clock in preparation for jammie time. Elmo welcomes Olivia Wilde for a game of "Teddy Bear Dress Up." Musical guest Sara Bareilles brings out two of her besties -- plus an extra special Sesame Street friend -- to help her perform her very own version of "Twinkle Twinkle Little Star."

Jason Sudeikis / Ciara

Elmo isn't shy about an important part of his bedtime routine -- potty time; after he challenges Jason Sudeikis to a round of the "Silly Dance Game," Ciara stops by to share an inspiring musical message; Grover shakes things up as he plays usher.