To Read and to Feel: The Multimodal Representation of Emotions in Digital Children’s Literature

ABSTRACT

Digital literature presents new ways of constructing aesthetic and emotional experience for child readers. This article reflects on emotion and affect in digital literature through the analysis of three literary apps for children. To deal with the complex phenomenon of the affective and emotional experience in digital reading, an interdisciplinary theoretical framework is proposed, relating Merleau-Ponty’s phenomenology to Kress and van Leeuwen’s multimodal social semiotics. The corpus includes the apps Little Red Riding Hood, by producer Nosy Crow, The Monster at the End of This Book, by Jon Stone and Michael Smollin, and Hat Monkey, by Chris Haughton. It discusses how different semiotic resources are used in the representation of emotions and in suggesting emotional tension to the child reader, with embodiment and the intrinsic relationship between reader’s bodies and their emotional experience as a central part of apps’ textual construction.

KEYWORDS:
Digital children’s literature; Literary apps; Emotions; Multimodality; Phenomenology

RESUMO

A literatura digital articula novas maneiras de construção de uma experiência estética e emocional aos leitores na infância. Este artigo reflete acerca da emoção e da afetividade na literatura digital por meio da análise de três aplicativos literários voltados a crianças. Para lidar com o complexo fenômeno da experiência afetiva e emocional na leitura digital, um referencial teórico interdisciplinar é proposto, relacionando a fenomenologia de Merleau-Ponty à semiótica social multimodal de Kress e van Leeuwen. O corpus inclui os aplicativos Little Red Riding Hood, da produtora Nosy Crow, The Monster at the End of This Book, de Jon Stone e Michael Smollin, e Macaco de chapéu, de Chris Haughton. A análise indica como os diferentes recursos semióticos são utilizados na representação das emoções e na sugestão de tensão emocional ao leitor infantil, tendo como centralidade a questão da corporeidade e a relação intrínseca entre os corpos e a experiência emocional.

PALAVRAS-CHAVE:
Literatura infantil digital; Aplicativos literários; Emoções; Multimodalidade; Fenomenologia

Introduction

The aesthetic experience of literature is deeply related to its ability to move the reader. Through a set of codes in various semiotic systems or modalities, literature provokes us, moves us and teaches us ways to feel. Children’s literature today presents a great diversity of formats and media, using different languages, semiotic modes and materialities to build its aesthetic potential. Literary apps, interactive and multimodal digital literary works available as apps for mobile devices, are a manifestation of contemporary literature, but they have not received much emphasis on children’s literature research. Thus, this study aims to contribute to studies of the poetics of literary apps by reflecting on how the multimodal and interactive nature of these works is articulated in the direct representation of emotions, but also in the construction of situations, scenes and narratives that emotionally affect the reader.

Through a survey of more than a hundred literary apps selected on the basis of awards and recommendations from magazines and specialized websites,¹ three apps were chosen for analysis: Little Red Riding Hood, by Nosy Crow (2013); The Monster at the End of This Book (hereafter Monster) (2011) by Jon Stone and Michael Smollin; and Hat Monkey, by Chris Haughton (2013). The criteria for selecting the corpus considered apps with high aesthetic potential and literary value, which used the possibilities of the digital medium - in this case, multimodality and interactivity - in their literariness. The apps present three distinct possibilities for the creation of emotional involvement between the reader and the narrative: in Monster, the reader must overcome their fear of the monster at the end of the book in order to continue with the story; in Little Red Riding Hood, the reader, along with the protagonist, is brought face to face with danger, manifested by the Big Bad Wolf; in Hat Monkey, the reader has the power to communicate with the protagonist and, depending on the content of the message, provoke emotions, such as joy, sadness, affection and care. Thus, the selected works are rich cases for analyzing the relationship between affect, emotion and embodiment in the literary experience.

1 Emotion, the Senses and Literature

In the original context attributed by Alexander Baumgarten (1993)² in the 18th Century, aesthetics refers to knowledge acquired through the senses, related to phenomenological experience as later presented by Maurice Merleau-Ponty. From this perspective, the body is inseparable from aesthetic knowledge, strongly related to affect and emotions, being the medium through which experience and thought take place. The emotional experience of reading is therefore an essential dimension of the literary aesthetic experience. A quality literary text uses its narrative and formal properties to provoke an emotion in the reader, to affect them. According to Merleau-Ponty,

To be emotional is to find oneself engaged in a situation that one is unable to cope with and yet from which one does not want to escape. Rather than accepting failure or retracing his steps, the subject abolishes the objective world that blocks his path in this existential dilemma and seeks a symbolic satisfaction in magical acts (2012, p. 88).³

Thus, being moved by literary reading is deeply linked to the pleasure of the text itself and to the raison d’être of literature, in its function of provoking through internal displacements (often uncomfortable for the individual). The emotional experience of literature is therefore deeply related to the proposal of defamiliarization orostranenie suggested by Viktor Shklovsky (1971).⁴ The existential impasse caused by the emotion experienced in literary reading is responsible for prolonging the duration of perception, which Shklovsky considers as the essential function of art.

Emotion derives from the concept of affection, which, according to the philosophy of Spinoza, designates “the effect that one body produces on another” (Brazão, 2018, p. 80)⁵ in a mutual relationship that defines one as the affecting body and the other as the affected body. According to Muniz Sodré (2006), affect is “a change of state and tendency towards a goal, provoked by an external cause” (p. 28),⁶ and emotion, a derivation of affection, derives from the Latin emovere, which refers to an “energetic or spiritual ‘movement’ from a zero point to an original point in the direction of another, as a consequence of a certain tension, capable of organically affecting the human body” (p. 29).⁷

The question of the reader’s movement, both internal and external, is a fundamental point of reflection in the aesthetic experience promoted by literary apps. In traditional literary reading, the text most commonly acts as an affecting body and the reader as the affected body. In digital literature, however, interactivity allows these roles to be switched. The reader gains the possibility of action, the possibility of affecting the text, although the ways in which the text can be affected are determined by the text itself. In this way, just as the text can provoke emotions in the reader, the reader can be given the possibility to determine certain emotions experienced by a character. If emotion presupposes this inner “movement” of the reader, what happens when the reader’s body is actually moving and the result of this movement, of their actions, simultaneously “provokes emotions” in the fictional characters?

Affect occurs through the senses, and is related to the individual’s body and phenomenological experience. According to Merleau-Ponty (2012), “when a text is read in front of us, and if the expression is successful, we do not have a thought on the margins of the text itself. The words occupy our entire mind” (p. 220).⁸ We experience literature while we are reading, immersed in the fictional universe, and we are emotionally affected by the literary aesthetic experience. The emotion experienced is not only the result of the perception that takes place through the reader’s body, but the emotion itself is a change in the state of that body.

[...] the emotion, as a variation of our being in the world, is already contingent with regard to the mechanical resources contained in our body, and manifests the same powers of articulating stimuli and situations that are at their height in language (Merleau-Ponty, 2012, p. 194).⁹

This intrinsic relationship between emotion and the body becomes an important object of reflection in the analysis of digital literature since the engagement of the body in reading literary apps is altered both in frequency - the body is systematically required to carry out the interactions -, and in form - the reader not only engages the body in trivial movements, as defined by Espen Aarseth (1997) such as turning pages, but rather performs a series of movements, often different for each scene, involving touch, but also more performative expressions such as blowing or shaking the screen.

Affect and emotion emerge from phenomenological, pre-reflective experience. Following the framework proposed by Charles Sanders Peirce (2005),¹⁰ a perception promotes awareness in Firstness, an instantaneous apprehension prior to interpretation and the creation of symbols. In Secondness comes emotion, or as a reaction to the quality experienced in Firstness. However, affect and emotion also depend on the semiotic relationships established in the multimodal narrative, in Thirdness. According to Merleau-Ponty, “there is a unity of the imagination and of the understanding, [...] and that, in an experience of beauty, for example, I undergo the experience of a harmony between the sensible and the concept” (2012, p. xxxi).¹¹ Therefore, phenomenological perception and logic (semiotics) are two essential facets of this phenomenon.

According to multimodal social semiotics, meanings are constructed in unified wholes that articulate signs in multiple semiotic modes. In the case of literary apps, these modes can involve written and spoken language, on the visual level; static and moving (or kineiconic) pictorial languages; on the sound level, in addition to the voice, we have music and sound effects; finally, there is the gestural level, involved in interactive actions. According to Gunther Kress (2010), each semiotic mode engages different aspects of perception. In literary apps, written verbal language and illustrations engage the sense of sight; animation engages both sight and hearing, with music and sound effects, as well as the oral speech/narration of the story, while interactive gestures engage the sense of touch and can explore the spatiality of the reader’s body. These modes use the potential and limitations of these media to communicate literature.

2 The Multimodal Representation of Emotions in Digital Children’s Literature

The multimodal nature of digital children’s literature is a fundamental aspect in the study of the representations of emotions in literary apps, since each semiotic mode has the potential to represent emotions or aspects of emotions in a different manner, according to their affordances (Kress, 2010), or their possibilities and limitations of representation. These representations, however, are unified in integrated multimodal ensembles, and therefore, it is within the synergy between them that representations of emotions will be constructed, which aim to affect the reader and thus, build their aesthetic experience.

The issue of multimodality in the representation of emotions in children’s literature was explored by Maria Nikolajeva (2012). The author highlights how, in picturebooks, images play a fundamental role in the representation of emotions, which is done both directly, through the facial and bodily expressions of the characters, and metaphorically. However, the literary apps selected present an even greater semiotic complexity than that of picturebooks, with the use of sound, animated images and interactivity (which includes the reader’s gestures), expanding the possibilities for representing emotions both diegetically - as experienced by the characters - or extra-diegetically, creating a general atmosphere aimed at provoking emotions in the reader. Among the many possible semiotic modes, however, each literary work selects the modes and explores different representational possibilities within a myriad of options. Thus, the three apps analyzed in this study present very different strategies in the multimodal representation of emotions, but in all cases, it is in the synergy between the semiotic modes, exploiting the affordances of each of them, that a multimodal literary text is successfully constructed, a text which moves the reader in some way, allowing them to experience emotions as a result of the poetic experience lived.

2.1 Pictorial Representation: Still and Animated Images

As Nikolajeva pointed out with regard to picturebooks, pictorial language plays a central role in the representation of emotions in literary apps. This is because it is through illustrations that the characters’ faces and bodies are represented, indicating their moods and emotions. Facial expressions are considered key to the expression and reading of emotions - to the point that the inclusion of these expressions in inanimate objects automatically humanizes them, transforming them into animate beings and therefore, worthy of empathy (Colombetti, 2014). In literary apps, the movement of images is also important in these representations. Firstly, by indicating changes in the emotions of the characters as the story unfolds, and secondly, by allowing the representation of the movements of their bodies to suggest their emotional states (trembling, for example, can signify fear).

However, this resource was only used to its full potential in one of the works analyzed, Monster, which details the protagonist’s facial expressions, portraying the fluctuations in his emotional state throughout the story. But why didn’t the other works use this resource, which is so common in picturebooks? The analysis of these works and their technical achievements suggests two reasons: (1) because of the representation styles of the illustrations, which opted for high levels of simplification; and (2) because of the animation techniques used - 2D animations with a lower number of frames per second and therefore, less capable of reproducing all the nuances of the changes in facial expression. Animations of this type are easier and cheaper to make, resulting in faster production and a more affordable cost, given that the complexity of developing these works makes them expensive to produce and yet, without a captive audience to guarantee a financial return.

In Little Red Riding Hood, we have human characters (Little Red Riding Hood, her mother and grandmother) and anthropomorphized animals (the Wolf and other animals she meets along the way). The latter have all the main facial elements that make it possible to express emotion in a way that is recognizable to humans, such as a mouth, eyes and eyebrows (according to seminal research in this area, such as that by Paul Ekman and Dacher Keltner (1997)). However, the use of facial expressions and modulations throughout the story is limited in this app. For most of the story, the characters show relatively neutral expressions of emotion with little variation (refer to Figure 1). When their facial expressions are explored in an expressive manner, in most cases they reveal more of the characters’ personality traits - such as the Wolf being evil and Little Red Riding Hood being brave - than their emotional states, although the two cannot be completely dissociated.

Figure 1
Screenshots of three moments from the Little Red Riding Hood app. Despite the emotional charge of the scenes, the characters’ facial expressions remain the same. © Nosy Crow and Ed Bryan. Source: Nosy Crow; Bryan (2013).

In the case of Monster and Hat Monkey, we have two anthropomorphized protagonists, whose faces have reduced expressiveness, due to a more abstract and simplified style of representation (as opposed to a realistic style, as discussed by Scott McCloud (1994)). In the case of Hat Monkey, the character doesn’t even have a mouth, relying exclusively on eye movements to express emotions. Despite these limitations, in certain scenes, eye movement is used expressively to indicate his emotions. For example, in the scene where the reader has to send a text message to the Monkey, he or she can choose between a happy message (( ), a sad one (( ), a message of love or affection (a heart) or a banana, which can also mean affection and care (refer to Figure 2). When he receives the happy message, he responds with joy, represented especially by the expression of his eyes, which contract, and his arms, which are raised in a celebratory manner; in response to the sad message, the Monkey shows a sad, almost tearful look. In the heart message, he hugs himself and closes his eyes tenderly, expressing his pleasure at being loved by the reader. Although the app does not use these facial expressions in any significant way, when they occur at specific moments in the story, image, movement and sound are successful in representing emotions. However, these visual expressions of emotion are always accompanied by vocalizations, sighs emitted by the character that confirm the expression of these feelings. It is within the multimodal ensemble that the meaning is completed for the reader.

Figure 2
Screenshot of the Hat Monkey app, which simulates a smartphone interface, where the reader can send a text message to the Monkey. © Fox and Sheep. Source: Houghton (2013).

Figure 3
Screenshots of the Hat Monkey app. The Monkey’s reaction to each of the possible messages: happy face, sad face, heart and banana respectively. © Fox and Sheep. Source: Haughton (2013).

Monster is undoubtedly the most successful app in the visual representation of emotions within the study’s corpus. It makes use of movement and animation resources, although the characteristics of the character’s illustration can be limiting in terms of the representation of facial expressions. Grover, the protagonist, is a puppet and his mouth, for example, has limited movements. However, the app has an important emotional charge; after all, it deals with the protagonist’s fear of the possible monster at the end of the book, and therefore, it uses other resources to represent Grover’s emotions: it makes very expressive use of the character’s gaze, movements and body expression, especially his arms and their positioning in relation to his head (refer to Figure 4). Grover is also always moving, he is very agitated, and this movement is an elaborate way of representing his anxiety. His exaggerated movements, on the other hand, disqualify him as a narrator, contributing to an ironic construction of the text and allowing the reader to ignore his requests not to turn the page.

Figure 4
Screenshots from the Monster app. This sequence shows the highly expressive way in which the representation of emotions through visual resources of facial and body expressions is used by the literary work. © Sesame Street. Source: Stone; Smoolin (2011).

2.2 Voiceover Narration

In Monster and Little Red Riding Hood, the voiceover narration stands out among the modes of emotional expression. The voice, a sound created by the expression of the speaker’s body, vibrates in tune with their emotions. Thus, the human voice is hardly neutral when it comes to emotions, and recognizing a sad, excited or frightened voice, for example, is immediate. Babies, even before they are able to understand verbal language, respond to the tone of voice of their caregivers, highlighting the voice’s potential for meaning and representation of emotion. In addition, the voice appears in consonance with the meanings of verbal language and this coupling results in an even more explicit identification of the emotions imbued in speech. This resource, due to its power - but possibly also because it is easier to reproduce in these works (compared, for example, to an elaborate animation, as discussed above) - is widely used and effective in the works in question.

In Little Red Riding Hood, for example, during the story’s climax, when the girl arrives at her grandmother’s house and finds the Wolf impersonating the old woman, we have the famous dialog in which the girl is surprised by the woman’s appearance. This moment precedes the Wolf’s revelation and attack. While the images make no effort to show the girl’s surprise (refer to figure 1, third frame), the narration, with great emphasis and expressiveness, performs this communicative function. In Monster, on the other hand, the entire text is in direct speech and each line spoken by the protagonist is full of expression and emotion. The character’s despair and fear at every turn of the page are marked by screams and vocal exclamations. The expressiveness of the spoken verbal discourse is similarly highlighted by the typography and other graphic resources, which emphasize the words spoken most emphatically, such as the word “monster,” and graphically simulate Grover’s emotional state (refer to Figure 4). The intensity of the speech is closely linked to the intensity of the character’s facial and body movements. In synchronicity, words that receive greater emphasis are accompanied by tremors or prominent movements of his eyes or arms.

In Hat Monkey, there is no voiceover narration feature in the app. It is therefore up to the reader or their co-reader to provide this narration, either aloud (more likely in the case of shared reading) or silently (more likely in autonomous reading). Given the emotional charge of the voice and the spoken language discussed above, the transfer of this function to a co-reader also results in a transfer of these possibilities for representing emotions to another body, which can even use his or her own facial and body expressions as a form of emphasis, adding elements to these representations. Shared reading is highlighted as a moment of great exchange of affection and emotional involvement between adult and child, and the use of the voice with its potential for representing emotions can be an important contribution to this process. It’s worth noting that the option to disable voiceover narration is also present in the Little Red Riding Hood app, but not in Monster.

2.3 Written Verbal Language

In terms of verbal language, Monster is the only app that explicitly uses words to name the character’s emotions. Since the fear experienced by Grover is a central aspect of the narrative, the verbal text repeats this information twice. Right at the beginning of the narrative, it is revealed that there will be a monster at the end of the book and Grover says, “Oh, I’m so scared of Monsters!!!.” The punctuation emphasizes the intensity of Grover’s emotion as he utters these words. In the scene before the end of the story, when the monster is revealed, Grover repeats, “Oh, I’m so scared!” as he begs the reader not to turn the page. At the end of the story, when it is revealed that he, Grover, is the monster, the character says, ironically, that the reader was the one who was scared: “And you were so scared.” In this scene, the dialogic nature of the narrative is explored ironically and playfully, implying a certain emotional response from the reader, which may or may not match reality. In Frederico (2018), a few readers showed clear signs of apprehension at the possibility of the monster, while others treated the situation as comical, dealing with it humorously.

In Little Red Riding Hood, the verbal text follows a traditional structure, with a third-person narrator whose indirect discourse alternates with moments of direct discourse. In this app, the characters’ emotions are not named or indicated directly by the verbal text. It is up to the reader to interpret the narrative as it unfolds in order to identify possible emotional states of the protagonist and other characters. In the dialogues, there may be some indirect indication of their emotional state. For example, Little Red Riding Hood might say in certain scenes, “What a lovely day!” or, “These trees are lovely!,” which indicate her appreciation of the situation, reverberating a sense of contentment. It’s worth noting that these phrases appear randomly and only when activated by the reader through interaction. Therefore, they play a complementary role and are not part of the core of the narrative; it is up to the reader to access these layers of meaning and not all readers will do so.

In Hat Monkey, the verbal text has a structure that breaks away from traditional forms: questions are addressed to the reader asking if they can perform a certain action with or for the Monkey. There is no clearly defined narrative, but a sequence of interactive scenes representing the character’s everyday experience in his interactions with the reader. Given this lean textual structure, there is no verbal reference to the character’s emotional states. When we move on to the next screen, visual and sound elements are predominant; we are immersed in the Monkey’s world (see Figure 5). The Monkey “speaks” in certain scenes, such as when the reader calls him on the phone, but only gibberish, in which words cannot be identified. The prosody and intonation of his speech, however, indicate his emotional state, reinforcing the expressive power of the spoken word, in this case, to the detriment of the semantic aspect of the text.

Figure 5
Screenshots from the Monster app. The first screen shows the verbal text. The reader then moves on to the visual narrative. © Fox and Sheep. Source: Haughton (2013).

2.4 Music

Music can be used in a fundamental way to evoke emotions in works that allow the use of this resource. Movies are a form of artistic expression that throughout its history has developed ways of expressing emotions in narratives through music, and various literary apps use music in a similar manner. All three works analyzed use music as a narrative resource at some point in the narrative, but it is certainly in Little Red Riding Hood that this resource is explored most explicitly with the function of suggesting emotions to the reader.

Throughout a significant part of the app, music is an important element in the setting of the story, suggesting a distinct emotional tone at each moment of the narrative; the musical variation marks changes in the course of the story. For example, while the protagonist is at home, we have a cheerful waltz; when she goes into the woods, the reader is invaded by somber music. This change in tone caused by the change in music is dramatic and generates great suspense. In this scene, the girl is walking alone through the dark woods and the Wolf, hidden among the vegetation, can be spotted by an attentive reader. Little Red Riding Hood, however, is unaware of the Wolf’s presence and continues to walk calmly through the woods.

The non-diegetic music has, therefore, the role of provoking emotions in the reader, independently of the emotions experienced by the character. It’s not a question of feeling what the character feels, but of fearing for her, as an observer. Throughout the app, Little Red Riding Hood expresses no fear of the wolf. The little girl is always confident in the face of the threat, whether in her facial expressions or her tone of voice, but the app’s music highlights the great danger she is experiencing and can thus arouse apprehension and even fear in the child reader.

2.5 Sound Effects

Sound effects can be used in a significant manner to suggest emotions in literary apps, but that doesn’t mean that all works use this resource for this purpose. Hat Monkey has sound effects, but these are generally used to create a sound environment that makes the story’s settings and actions more realistic, contributing to verisimilitude and for the reader’s immersion. This use of effects can also be seen in the other works. In Little Red Riding Hood, however, we have sound effects in the most dramatic scenes, which contribute to the atmosphere of danger and suspense in the story. These scenes can evoke emotions in the reader, as discussed by Giselly Lima de Moraes (2016). This is the case of the scene described above, of the girl walking through the woods, which, as well as having somber music, features the sounds of birds and small animals. These sounds create the sonic ambience of the woods that envelops the reader, creating a feeling of deep immersion into that fictional space. These are diegetic effects and place the reader and the character in the same environment. Thus, we have sound modalities, effects and music, acting in a complementary manner, but overlapping different levels of diegesis. Moraes also points out that, in the scene of the famous dialogue, the music stops and becomes an effect, a sound of great suspense, created by the frantic friction of the bow against the strings of a cello. This movement reverberates, by means of the sound, through the reader’s body and can create great apprehension.

Monster is the app that uses sound effects throughout the narrative in the most emphatic and sophisticated way, referring to the tradition of children’s cartoons, in which effects are widely used. In the case of representations of emotions, the effects serve as a basis or support for the emotional tension generated by the character’s speech, helping to add intensity to the vocal expression. The most obvious example is when the word “monster” is uttered - and this word appears in every scene of the app. In addition to Grover’s trembling voice, a terrifying keyboard sound is triggered, accompanied by a tremor in the word written on the screen, which becomes slightly deformed. Therefore, we have an orchestration of verbal, visual and sound effects that emphasize the terrifying aspect of the monster, indicating the character’s fear and suggesting the same emotion to the reader.

Another notable example occurs in the scene before the monster is revealed, where the sound effects make a significant contribution to building an atmosphere of danger, in line with the fear expressed by Grover through his speech and body language. Here, the sound effect is non-diegetic, but the emotion it seeks to provoke in the reader coincides with the emotion experienced by the character, unlike the Little Red Riding Hood example discussed in the previous section, in which the music aims to provoke apprehension in the reader, even though the character is walking calmly through the woods.

2.6 Interactivity

Interactivity is a fundamental characteristic of digital literature, which allows the reader to participate in a variety of ways in these literary texts. Through the actions carried out by the reader, the text takes on different configurations, allowing a variety of reading paths and experiences. The construction of meaning through interactivity is complex and multifaceted; this can be said about the ways in which emotion and interactivity are connected in digital reading.

The very presence of interactivity affects the emotional experience of reading, because the act of interaction itself is potentially charged with emotion for the reader. The act of interacting can be seen as potentially generating emotions; the possibility of intervening in the story and seeing the text transform as a result of the choices can add excitement to the act of reading, driven by the expectation of a response from the app. Thus, we can conclude that interactivity, as an element of the discourse (Chatman, 1978) of the digital narrative, has the potential to generate emotions, regardless of whether these actions represent some kind of emotion in the narrative.

The reader’s emotional response can be exacerbated depending on the context created by the narrative and the meanings inflicted by the reader’s participation in the story. In addition, interactivity brings an important facet to the analysis of representations of emotions: in interactive works, the reader’s body is an element of the multimodal text and a potential resource for the representation of emotions in the works through interactive gestures.

With each interaction, while reading a literary app, the reader can take on a role in the narrative, a role that is associated with certain emotions. In Hat Monkey, there are two main types of interaction. In the first group, the reader helps the Monkey, or performs actions for him. The reader opens the door for the protagonist to enter, feeds the Monkey by giving him bananas (the bananas are out of reach), makes a phone call to the character, texts him on his phone and, finally, the reader turns the pages of the book that the Monkey is reading and turns off the light in his room so that he can sleep. In another group of interactions, the reader and the Monkey perform activities together, they play hide and seek, do a “high five,” dance, play music and talk on the phone.

Emotions are associated with these actions on two levels. On the first level, the emotions result mainly from the emotional impact that the actions have on the character. The reader feeds the Monkey, who is happy to eat the banana (which can be seen in his facial and body expressions); as a result, the reader is also expected to feel a sense of satisfaction and happiness, not from the act of picking up the banana and bringing it to the Monkey himself, but from the pleasure of having generated this positive emotion in the character. The reader takes care of the Monkey, doing something good for the protagonist and, as a result, feels good as well, in a move that promotes the development of an empathetic relationship between the reader and the character.

In the second group, the origin of the emotions is twofold: the reader’s action - dancing, for example - can generate a positive emotion in the reader - because dancing is supposed to be pleasurable. In addition, the reader can feel emotions - presumably positive ones - for sharing this moment with the Monkey and perceiving positive emotional reactions in him. The reader’s actions, therefore, do not directly represent an emotion, but the action is associated with certain emotions. The actual representation of the emotion - which occurs through the reader’s body showing joy or fear or sadness, for example - will depend on the reader and the repertoire, expectations and motivations they bring to the reading. A shy child, for example, may feel uncomfortable with the need to dance with the Monkey in front of their classmates, in a shared reading situation at school, for example. Thus, the representation of emotions through interactivity carries a high degree of indeterminacy and is only effectively achieved while reading.

Monster is another interesting example when discussing this imprecision in the representation of emotions through interactivity. The app ironically suggests that the reader contradicts Grover’s requests not to turn the page, so as not to get to the monster at the end of the book. The character asks the reader not to turn the page, but the expectation is that he will. As discussed in Frederico (2017) the multimodal construction of the story suggests a courageous identity to the reader; while Grover is afraid, the reader is invited to be brave, but the emotional experience that this act will generate in the reader may vary greatly. One possibility is of a reader who isn’t afraid of monsters and finds Grover’s reaction exaggerated and caricatured; the resulting emotion could therefore be one of humor, laughter. Another possibility is of a reader who is not afraid of the monster, but empathizes with the character - Margaret Mackey (2016) talks about the case of a child who decided not to continue with the story, in response to Grover’s request. It is also possible that other readers feel afraid of the monster, but keep turning the page anyway and keep following the story (as reported in Frederico (2021)). These are merely a few of the multiple emotional responses that interactivity in an app such as Monster can generate.

In Little Red Riding Hood, we have a different scenario. In many of the interactions, the reader takes on the role of the protagonist. When the reader becomes Little Red Riding Hood, they immerse themselves in the fictional world, experiencing the narrative in the first person. Thus, the character’s emotional state when carrying out that action is transferred to the reader, who experiences it firsthand, through his or her own body. When the reader “attacks” the Wolf (for example, by throwing a jar of honey at him), the intention is that they experience first-hand the eagerness to get away from the Wolf and the impetus that leads to the action, in a mixture of fear and courage. This type of interaction, therefore, can allow the reader to go beyond empathy - when, despite understanding the character’s feelings, they still see him as someone else - to truly experience identification, which occurs when the reader actually feels like he’s walking in the protagonist’s shoes (Mar et al., 2011). Not only does the reader experience the tension of the situation firsthand - although the level of performance varies according to the gesture used -they can also use the intensity of the gesture to express the emotion experienced, thus enabling another form of reverberation of that emotion, amplifying it.

The gestures that allow interaction with the story are part of the multimodal configuration of literary apps (Frederico, 2018). The form of these gestures (tapping, swiping, shaking the device, etc.) forms part of the multimodal text, representing aspects of the narrative and suggesting meanings. Here, there’s an important indeterminacy as well. The signs created through these gestures are merely suggestions of the app, and their final form will once again depend on the choices made while reading. The reader can activate interaction with the app through gestures that diverge from what is suggested. By improvising other gestures, new meanings are generated. Because this article does not consider reader responses, the following analysis will focus on the suggested gestures proposed by the works and how their forms represent emotions in the narrative.

An example of this way of representing emotions using the reader’s body through interactive gestures can be seen in Little Red Riding Hood. After the Wolf’s attack, who jumps out of bed and goes after the girl to devour her, Little Red Riding Hood and the Wolf find themselves opposite each other around the dinner table. The reader must then move Little Red Riding Hood around the table, helping the girl to escape. This scene has very tense music and the frantic movement of the reader can thus act as a way of representing the fear and tension of the moment. The reader can only move the character in circles and cannot remove her from that situation, which also has the potential to exacerbate the emotional charge of the scene. Music here creates, along with the gestures, an important multimodal ensemble in creating this representation.

The analysis of the apps in question, however, indicates that the emotion in these cases results mainly from the context in which the gesture was made, its meanings and the relationships it establishes between text and reader, and less from its form, which plays a representational role in the narrative. Thus, in most cases, the works in question did not make extensive use of interactive gestures as a resource for representing emotions and the emotional expressiveness of the gesture is provided from the readers themselves, who can add vigor, speed or hesitation to the gesture and thus express their emotional state through it.

Final Considerations

The analysis of the literary apps in the corpus reveals how the representation of emotions in digital children’s literature articulates the various semiotic modes in different ways, generating different multimodal configurations. These multimodal representations are articulated in two ways: in the first, the emotional experience is evoked from the representation of the characters’ emotional state, thus seeking to provoke empathy in the reader and affect them emotionally in an indirect manner - since having empathy does not necessarily imply feeling the emotion experienced by the character (Feagin, 2019 apudNikolajeva, 2012); in the second way, the literary work evokes certain emotions directly in the reader, regardless of the character’s emotional state. These two levels are directly related to the constitutive elements of the narrative: the story and the discourse (Chatman, 1978). In the first case, the emotional experience is proposed from the story that is being told, while in the second construction, the emotion is suggested from the discourse level, or how that story is being told. In the same way that story and discourse are inseparable in narrative, in the representation of emotions, these two layers are articulated to suggest a certain emotional experience that is unified with the literary text.

The concept of “modal intensity” proposed by Sigrid Norris (2004), allows us to reflect on the role of these different semiotic modes in the representation of emotions in literary apps, and to relate this literary form to other forms of multimodal narrative for children. Modal intensity refers to the intensity or weight that a semiotic mode assumes in a multimodal ensemble (Norris, 2004). The intensity of each mode varies at each moment of multimodal communication and certain semiotic modes are more important in the multimodal ensemble than others. Thus, looking specifically at the representation of emotions in the narratives, and considering the representational possibilities of each of these modalities, we find that those of an auditory nature - music and the narrator’s voice - have a high intensity in the multimodal ensembles. The sound resonates in the readers’ bodies, affecting them directly and powerfully, often still in Firstness. Given the diffuse and subjective nature of emotions in literary texts, voice and music have an expressive potential that allows emotions to be represented poetically in line with the narrative.

In certain works (such as Monster), the aural modalities can also be closely accompanied by animation, in which the characters’ emotions are represented especially through facial and body expressions. Narration and animation show significant concurrence in the representation of emotions, intimately complementing each other. However, we have also seen that, in Hat Monkey and Little Red Riding Hood, animation is not always used to its full potential in the representation of the characters’ emotions.

The interactive aspect of these works also presents new possibilities for the emotional experience with a literary text. The very possibility of participation can generate emotions in the reader, who has to intervene in the narrative and can affect it. By interacting, the reader can also take on the perspective of a character, resulting in an identification with the character that amplifies the emotional experience due to its immersive quality.

The analysis of the ways in which emotions are represented and suggested in the selected apps’ texts, has revealed how its multimodality is explored in a complex and distinctive way in each work. Embodiment, however, is the central axis and guiding thread. On the one hand, the representations of the characters’ emotions are always deeply linked to that character’s body, either through the representation of their facial or bodily expressions, or through the resonance of such an emotion through their voice. In aural modalities, the reader’s body is immersed in the sound space promoted by the narrative, and vibrates according to the intensity of the sound. The reader can feel this vibration and the emotions it carries in the flesh. Interactivity, on the other hand, can put the reader inside the narrative, performing certain actions from the character’s perspective, and thus, their body and the character’s body merge, making it possible to experience these emotions first-hand, in a type of immersion and participation in a fictional universe that is typical of digital reading.

The multimodal analysis of the representation of emotions in literary apps, therefore, suggests a return to the centrality of the body in the aesthetic experience, as proposed by Baumgarten¹² and Merleau-Ponty,¹³ but also reveals new ways in which digital, multimedia, multimodal, interactive works of art can move readers and promote aesthetic literary experiences for children and young people today.

REFERÊNCIAS

• AARSETH, Espen J. Cybertext: Perspectives on Ergodic Literature. Baltimore, Md: Johns Hopkins University Press, 1997.
• BAUMGARTEN, Alexander. Estética: a lógica da arte e do poema. Tradução de Míriam Sutter Medeiros. Petrópolis: Vozes, 1993.
• BRAZÃO, José Carlos Chaves. Afecção e afeto em Spinoza e Daniel Stern. Ayvu: Revista de Psicologia, v. 4, n. 2, pp. 77-95, 2018. Disponível em: https://periodicos.uff.br/ayvu/article/view/22240 Acesso em: 28 abr. 2024.
  » https://periodicos.uff.br/ayvu/article/view/22240
• CHATMAN, Seymour Benjamin. Story and Discourse: Narrative Structure in Fiction and Iilm. Ithaca, NY; London, UK; Cornell University Press, 1978.
• CHKLOVSKI, Viktor. A arte como procedimento. In: TOLEDO, Dionísio de Oliveira (org.). Teoria da literatura: formalistas russos. Tradução de Ana Mariza Ribeiro. Porto Alegre: Editora Globo, 1971.
• COLOMBETTI, Giovanna. The Feeling Body: Affective Science Meets the Enactive Mind. Cambridge, Mass.; London, UK: The MIT Press, 2014.
• EKMAN, Paul.; KELTNER, Dacher. Universal Facial Expressions of Emotion: An Old Controversy and New Findings. In: ULLICA SEGERSTRÅLE; Peter Molnár (org.). Nonverbal Communication: Where Nature Meets Culture. Mahwah, New Jersey: Lawrence Erlbaum Associates, Publishers, 1997. pp. 27-46. Disponível em: https://www.jstor.org/stable/55476 Acesso em: 28 abr. 2024.
  » https://www.jstor.org/stable/55476
• FREDERICO, Aline. Children Making Meaning with Literary Apps: A 4-Year-Old Encounter with The Monster at the End of This Book. Paradoxa, v. 29, pp. 43-64, 2017.
• FREDERICO, Aline. Embodiment and Agency in Digital Reading: Preschoolers Making Meaning with Literary Apps. Orientadora: Dra. Fiona Maine. Co-orientadora: Dra. Maria Nikolajeva. 2018. Tese (Doutorado em Filosofia) - Homerton College e Faculty of Education, Universidade de Cambridge, Cambridge, 2018. 332 f. https://doi.org/10.17863/CAM.31007. Disponível em: https://www.repository.cam.ac.uk/handle/1810/283637 Acesso em: 28 abr. 2024.
  » https://doi.org/10.17863/CAM.31007. Disponível em: https://www.repository.cam.ac.uk/handle/1810/283637
• FREDERICO, Aline. Lendo um aplicativo: perspectivas da construção de sentido na leitura literária digital na primeira infância. Perspectiva, v. 39, n. 1, pp. 1-25, 2021. https://doi.org/10.5007/2175-795X.2021.e66013. Disponível em: https://periodicos.ufsc.br/index.php/perspectiva/article/view/66013 Acesso em: 28 abr. 2024.
  » https://doi.org/10.5007/2175-795X.2021.e66013. Disponível em: https://periodicos.ufsc.br/index.php/perspectiva/article/view/66013
• HAUGHTON, Chris. Macaco de chapéu. Fox & Sheep, 2015. Versão 1.8 (2017). Aplicativo disponível para sistema iOS Apple em https://apps.apple.com/br/app/macaco-de-chap%C3%A9u/id904565251 Aplicativo disponível para sistema Android em: https://play.google.com/store/apps/details?id=com.mobileventures.HatMonkey&hl=ptBR≷=US
  » https://apps.apple.com/br/app/macaco-de-chap%C3%A9u/id904565251» https://play.google.com/store/apps/details?id=com.mobileventures.HatMonkey&hl=ptBR≷=US
• KRESS, Gunther. Multimodality: A Social Semiotic Approach to Contemporary Communication. New York, NY; London, UK: Routledge, 2010.
• MACKEY, Margaret. Digital Picturebooks. In: ARIZPE, Evelyn; STYLES, Morag (org.). Children Reading Picturebooks: Interpreting Visual Texts. 2. ed. London, UK; New York, NY: Routledge, 2016. pp. 169-179.
• MAR, Raymond A.; OATLEY, Keith; DJIKIC, Maja; MULLIN, Justin. Emotion and Narrative Fiction: Interactive Influences Before, During, and After Reading. Cognition & Emotion, v. 25, n. 5, pp. 818-833, 2011. DOI: 10.1080/02699931.2010.515151. Disponível em: http://www.tandfonline.com/doi/abs/10.1080/02699931.2010.515151 Acesso em: 28 abr. 2024.
  » http://www.tandfonline.com/doi/abs/10.1080/02699931.2010.515151
• MCCLOUD, Scott. Understanding Comics: The Invisible Art. New York, NY: HarperPerennial, 1994.
• MERLEAU-PONTY, Maurice. Fenomenologia da percepção. Tradução de Carlos Alberto Ribeiro de Moura. São Paulo: Martins Fontes, 1999.
• MORAES, Giselly Lima de. Trilha sonora de aplicativos para crianças e educação literária. 2016. Tese (Doutorado em Educação) - Faculdade de Educação, Universidade Federal da Bahia, Salvador, 2016. 246 f. Disponível em: https://repositorio.ufba.br/ri/handle/ri/20840 Acesso em: 28 abr. 2024.
  » https://repositorio.ufba.br/ri/handle/ri/20840
• NIKOLAJEVA, Maria. Reading Other People’s Minds Through Word and Image. Children’s Literature in Education, v. 43, n. 3, pp. 273-291, 2012. https://doi.org/10.1007/s10583-012-9163-6 Disponível em: https://link.springer.com/article/10.1007/s10583-012-9163-6 Acesso em: 28 abr. 2024.
  » https://doi.org/10.1007/s10583-012-9163-6» https://link.springer.com/article/10.1007/s10583-012-9163-6
• NORRIS, Sigrid. Analyzing Multimodal Interaction: A Methodological Framework. New York, NY: Routledge, 2004.
• NOSY CROW; BRYAN, Ed. Little Red Riding Hood. Nosy Crow, 2011. Versão 2.0.1 (2011). Aplicativo previamente disponível para iOS Apple em: https://apps.apple.com/us/app/little-red-riding-hood-nosy/id626696483?ls=1
  » https://apps.apple.com/us/app/little-red-riding-hood-nosy/id626696483?ls=1
• PIERCE, Charles S. Semiótica. Tradução de José Teixeira Coelho Neto. 3. ed. São Paulo: Editora Perspectiva, 2005.
• SODRÉ, Muniz. As estratégias sensíveis: afeto, mídia e política. Petrópolis: Vozes, 2006.
• STONE, Jon; SMOLLIN, Michael. The Monster at the End of this Book. Sesame Street, 2010. Versão 6.0 (2020). Aplicativo disponível para sistema iOS Apple em: https://apps.apple.com/us/app/the-monster-at-the-end/id409467802 Aplicativo disponível para sistema Android em: https://play.google.com/store/apps/details?id=org.sesameworkshop.monsterone.play&hl=en_US≷=US
  » https://apps.apple.com/us/app/the-monster-at-the-end/id409467802» https://play.google.com/store/apps/details?id=org.sesameworkshop.monsterone.play&hl=en_US≷=US

• Research Data and Other Materials Availability

  The contents underlying the research text are included in the manuscript.

• Reviews

  Due to the commitment assumed by Bakhtiniana. Revista de Estudos do Discurso [Bakhtiniana. Journal of Discourse Studies] to Open Science, this journal only publishes reviews that have been authorized by all involved.

• 1
  The full list of applications can be found in Frederico (2018), appendices section.
• 2
  BAUMGARTEN, Alexander. Reflections on Poetry: Alexander Gottlieb Baumgarten’s Meditationes philosophicae de nonnullis ad poema pertinentibus. Tradução de Karl Aschenbrenner e William B. Holther. Berkeley and Los Angeles: University of California Press, 1954.
• 3
  MERLEAU-PONTY, Maurice. Phenomenology of Perception. Translated by Donald A. Landes. New York: Routledge, 2012.
• 4
  SHKLOVSKIJ, Viktor. Art as Technique. In: RIVKIN, Julie; RYAN, Michael (eds.). Literary Theory: An Anthology. Malden: Blackwell Publishing Ltd, 1998.
• 5
  In Portuguese: “o efeito que um corpo produz sobre o outro.”
• 6
  In Portuguese: “uma mudança de estado e tendência para um objetivo, provocadas por causa externa.”
• 7
  In Portuguese: “‘movimento’ energético ou espiritual desde um ponto zero a um ponto originário na direção de um outro, como consequência de uma certa tensão, capaz de afetar organicamente o corpo humano.”
• 8
  For reference, see footnote 3.
• 9
  For reference, see footnote 3.
• 10
  PEIRCE, Charles S. The Essential Peirce, Volume 2: Selected Philosophical Writings (1893-1913). Bloomington: Indiana University Press, 1998.
• 11
  For reference, see footnote 3.
• 12
  For reference, see footnote 2.
• 13
  For reference, see footnote 3.

Review I

About the reviewer SCIMAGO INSTITUTIONS RANKINGS

Review I

The article is well structured and achieves its proposed objective, albeit without much depth. The paper’s objectives are clear and developed consistently in the text - it aims to contribute to our understanding of the poetics of literary apps - reflecting on how the multimodal and interactive language of these literary works is articulated in the direct representation of emotions or in the construction of situations, scenes and narratives that emotionally affect the reader. The theories chosen are current, but at the time of analyzing the Apps, there was no dialogue between theory and practice. I believe it is necessary to review and conduct this dialog, demonstrating how the theory helps in the analysis and interpretation of the Apps. The reflection had the potential of being original and contributing to the field of knowledge, but this reflection on theory and practice was lacking. The text is clear, correct and the language is appropriate for a scientific paper. It is well written, although there are a few comma slips and confusing phrases that are marked in the text so that they can be revised. The suggestions can be found in the opinion and in the body of the text and, if it is published, I suggest that you make the corrections, especially by dialoguing with the theories proposed and the analysis of the Apps. APPROVED WITH SUGGESTIONS [Revised]

• peer review recommendation: accept

History

• Peer review received
  24 Jan 2024

Review II

About the reviewer SCIMAGO INSTITUTIONS RANKINGS

Review II

The article presents a very interesting study proposal, as it provides aspects that are quite relevant to studies on multimodality, such as nonlinear/multidirectional reading, particularities related to music, voice and the effects of meaning resulting from sound, among others. It is also worth noting that the paper’s objectives of the work are in line with the development of the text, which enhances knowledge about multimodality and social semiotics. The references that support the study are quite relevant, especially when citing Kress (2010) and Norris (2004). The paper presents clarity, correctness and adequacy of language (there are small deviations that should be reviewed), which ratifies its originality, especially when dealing with the field of digital children’s literature. APPROVED WITH SUGGESTIONS [Revised]

• peer review recommendation: accept

Author’s Response

Author’s Response

https://orcid.org/0000-0002-5670-8548 Aline Frederico

I would like to thank the reviewers for their attentive reviews and for helping improve this manuscript. I have made the editorial changes suggested and added a paragraph at the end of the text to clarify the connection between the discussions in the first section, regarding aesthetics and phenomenology and the multimodal analysis of the text, clarifying therefore how the theories presented support the rest of the article. I believe this is what the first reviewer names “theory and practice.”

History

• Peer review received
  22 Apr 2024

Data availability

The contents underlying the research text are included in the manuscript.

Publication Dates

• Publication in this collection
  09 Aug 2024
• Date of issue
  Jul-Sep 2024

History

• Received
  10 Nov 2023
• Accepted
  13 June 2024