The following is a first draft introducing elements of the directed study analyzing positive sentiment towards female journalists on Twitter. This blog post focuses primarily on methods and approaches explored in the early stages of the research. Results will be posted in a separate blog.
Social media has vastly changed the way that the general public interacts with traditional media journalists. With the widespread use of social media platforms like Twitter, users can directly interact with journalists at lightning speed to comment on posts and share their views online. On average, there are approximately 500 million tweets posted on Twitter every day (Internet Live Stats, 2022), with each post reaching up to a maximum character limit of 280 characters (Rigolin, 2018). These tweets can include different kinds of attributes like hashtags and emojis, as well as additional attachments including photos, gifs, and videos. With the diverse amount of content available online, social media platforms face the challenge of moderating high quantities of this content on a daily basis.
Platforms have come to face the challenge of moderating online hate speech and inflammatory content posted online. In the process of regulating this content to protect freedom of speech, different models have been created to classify hate speech and moderate platforms to reduce harmful online activity. However, very little research has been conducted to identify the kinds of positivity that exist on online platforms. Research in this area is particularly important as it can help identify the kinds of interactions that exist and identify potential ways to promote positivity: for example, the way tweets are phrased to engage in meaningful dialogue and how to further engage users in positive discussions.
Furthermore, there is limited research comparing the differences in interpretation of positivity between machine learning models and human-coded sentiment. This case study aims to further understand the inconsistencies between tweets defined as positive by machine learning models versus positive by human coders. The research looks to respond to the following questions:
- What is the rate of inconsistency between human-coded tweets and machine learning in identifying positive tweets?
- What types of positive tweets were in fact labeled positive by machine learning?
- What attributes were most commonly found in individual tweets from the mislabeled positive tweets?
What Constitutes Positivity?
According to the Oxford Learner's Dictionary, positivity is defined as "the practice of being positive in your attitude and focusing on what is good" (Oxford Advanced Learning Dictionary, 2022). With the evolution of language, various cultural references, and openness to new ideologies over time, perceptions can drastically change regarding what constitutes a positive term or interaction over a period of time. Different terms can also take on new meanings with events that take place locally, nationally, or globally.
For example, before January 2022, the term "trucker" was generally viewed as a positive term referring to an individual delivering goods and services to Canadians. In January 2022, a trucker convoy headed to Ottawa, Ontario, and occupied the downtown core facing the Parliament of Canada for three weeks. This organized group looked to protest different government policies, including vaccine mandates, and attempted to force Justin Trudeau to resign as Prime Minister (Terrell, 2022). When looking at that same term following this event, more individuals see it in a negative light, especially if the individual was a resident of downtown Ottawa personally affected by the events on Parliament Hill. On the contrary, the term may also continue to be considered positive if the individual shares common ideologies with the organizers of the convoy. Positivity, therefore, can hold a subjective meaning and may not always be clearly defined, an additional consideration alongside more commonly associated terms like happiness, joy, words of encouragement, and compliments.
When referring to one-to-one human interactions, positivity is more clearly visible through the ways in which humans interact. It can often be measured through facial expressions and tone of conversation (Dael, 2012): smiling at an individual, providing words of support or encouragement, and meaningfully contributing to a neutral dialogue. Face-to-face interaction has the added benefit that non-verbal cues can reduce confusion or signal a need for clarity, and tone of voice offers implicit cues about sentiment. These cues aren't available online, which is what makes alternative means of understanding online sentiment necessary.
Cultural Bias
Another factor affecting sentiment analysis is the role cultural differences play in interpretation. Each individual's background and culture can greatly dictate how they interpret different symbols. For the Pixar film Inside Out, different regional versions of the film were produced that didn't change the storyline but preserved symbolic meaning across cultures. In one scene, a father tries to feed his one-year-old broccoli. In the American version, broccoli carries a negative sentiment as a commonly disliked vegetable. In the Japanese version, broccoli is popular, so it was replaced with green peppers to convey the same negative meaning to Japanese audiences (Acuna, 2015).
This kind of ambiguity may be less common with more traditionally used emojis in tweets, such as a thumbs up or a smiley face. But it does suggest that videos or images can carry different background meaning depending on the reader's perspective, as attachments can shift how the sentiment of a post is perceived.
Methods for Measuring Sentiment
Sentiment analysis, also commonly referred to as "opinion mining," is a classification system that assesses the overall sentiment of a fragment of content and attributes a polarity score. Scores are generally classified into three categories: positive, negative, or neutral (D. Sunitha, 2016), and the measurement can take a subjective meaning, as categories are determined and defined individually by the coder. This review of sentiment is especially important in Natural Language Processing (NLP), the field of computer science concerned with human-computer language interaction (D. Sunitha, 2016). Using sentiment analysis, training models are built to output a measurement of the sentiment of social media content.
There are numerous formulas for calculating sentiment. One approach counts the number of positive words minus the number of negative words (Philander, 2016): a 20-word post with 10 positive and 5 negative words would be classified as overall positive; more negative than positive words would classify it as negative; an equal split (e.g. "I hate Batman but love Superman") would classify it as neutral. Certain terms (like "and," "this," and "they") are removed from analysis entirely, since they carry no sentiment (Zavattero, 2015).
Marketers have made further advances in understanding sentiment in customer engagement, using a feature-level classification system with three tiers: document, sentence, and feature/attribute (Chiu, 2015). The document level classifies an entire piece of content as positive or negative based on its overall expressed sentiment, so a mostly positive post with a few negative terms would still be classified as positive overall. The sentence level determines sentiment within each individual sentence to weigh overall meaning. The feature level identifies opinion using specific attributes and features of the text, including lexical features (word or character-based measures like word length distribution and vocabulary richness), syntax (sentence structure and phrasing patterns), and structural features (organization and use of quoted content, such as retweets).
Punctuation also contributes meaningfully to sentiment interpretation. Exclamation points often signal positive tone, while some research suggests periods can read as negative in social media contexts. Quotation marks can imply sarcasm or added emphasis on a specific word, and capitalization can increase the perceived negativity of a post.
Context
Contextual analysis is another difficult consideration for sentiment analysis. Human-coded sentiment analysis is already time-consuming and labor-intensive, and social context, the visible information from a previous thread and list of interactions with other users (Sánchez-Rada, 2019), adds further complexity. Twitter is known for its limited, short word count, where much of a post's context is omitted in favor of alternative ways to connect posts, such as replying directly to create a thread or posting subtweets (Rigolin, 2018).
Sample
The sample of tweets analyzed in this case study was dated from August 5, 2019 to October 31, 2019, the campaigning period for the 2019 Canadian federal election. The collection included replies, mentions, quote tweets, and individual tweets directed toward users. The types of users in the dataset totaled 346 and included journalists, hosts, panelists, and editors.
The sample was collected through a multistep process. First, 743,685 tweets were run through BERT, an open-source machine learning model framework for natural language processing (Lutkevich, 2020). BERT's sentiment classification followed seven possible outcomes: highly negative, medium negativity, low negativity, positive, neutral, and unclear. Of these tweets, 29,319 were classified as positive by BERT. From that set, 1,000 were randomly sampled to complete the sentiment analysis.
Methods
The 1,000 positive tweets classified by BERT were manually coded using sentiment content analysis by a single human coder. The coding book was divided into three primary areas of analysis: human-coded sentiment, written content, and tweet attributes.
Human Coded Sentiment
Each tweet was classified under one of four categories: positive, neutral, negative, or undefined. The category was selected based on the term that most prominently defined the overall sentiment of the tweet. The fourth category, "undefined," accounted for tweets that didn't fit the first three, often because they could reasonably hold two sentiments and couldn't be assigned a single category. For example, a tweet reading "And they're friends with you so they can see what it's like when your team actually wins games" could plausibly be read as positive or negative without further context.
Written Content
Written content was defined as the subject matter of a tweet that could generate sentiment: the kinds of words used to develop a feeling. Because a single post could contain multiple sentiments, tweets were classified to include one or more categories to account for combinations of words. Eight categories were used to code written content: condemning hate, informative, apology, encouragement, compliment, negativity, swear words, and other.
Defined Positive Categories
- Condemning hate: a user directly responding to an account to acknowledge negative commentary directed at the user. ("Terrible. People can be so nasty.")
- Informative: any announcement of upcoming events, linked articles, or details shared with the public, excluding personal opinion. Statements from unverified accounts were excluded, since they couldn't be fact-checked. ("We are a few weeks away from an election, and I am really proud and excited to co-host Party from @cbcpodcasts with @RosieBarton.")
- Apology: use of "sorry" or "my apologies" directed at a user to demonstrate regretful behavior, whether for personal or another user's commentary, or for an outcome. ("@RVAwonk Yikes. Sorry you had to deal with that.") Classified as positive due to the implied regret and intent to reduce harm.
- Encouragement: words of motivation to continue an activity (a hobby, a project) or acknowledgement of a personal achievement. Terms included "congratulations," "great job," and "keep up the good work." Stand-alone terms like "good," "excellent," or "fantastic" without further context were also classified here rather than as a compliment.
- Compliment: a positive statement directed at a specific user. ("Wow, I watched the debate and have now read your piece. Fantastic work Mr. Tasker. I was eager to see CBC's reporting. I see fairness. Good work 🙂")
Defined Negative Categories
- Negativity: a formulation of ideas explicitly unpleasant to a user, or an opposing opinion that doesn't engage in constructive dialogue. ("Enjoy prison dipshit"; "@RosieBarton @CBCNews so nice to see you laughing about Alberta's issues. Wow!!! You should resign and go work for Justin where you belong.")
- Swear words: any coarse language not used in a professional context, including symbolic or partially obscured versions. ("OMFG"; "Congrats to every dumb F who voted UPC"; "Utterly f***ing amazing")
Additional Categories
Other was added to account for posts that didn't fit any of the categories above.
Tweet Attributes
The second part of the coding book assessed the format of each tweet to help identify patterns or inconsistencies in how sentiment was assigned. Seven attributes were measured:
- Emojis: any icon within the tweet text that could signal sentiment, including symbolic representations (e.g.
:)). - Punctuation: standard grammatical markers (
. , ; : ' ' " " ( ) -), which can meaningfully shift tone. Compare "Nice reporting" to "Nice “reporting”": the quotation marks in the second version can read as sarcastic or add emphasis. - Length: short (1–92 characters), medium (93–140 characters), or long (141–280 characters).
- Capitalization: none, partially capitalized, or fully capitalized.
- Attachments: additional content beyond text, such as images.
- Hashtags: whether the post included at least one hashtag (e.g.
#DefundCBC). - Language: English or French.
Limitations
Three limitations shaped this research. First, time constraints didn't allow for analysis of the full threads in which content was posted, context that could be particularly relevant for topics like politics and pop culture. Related to this, very short replies were difficult to assess for deeper meaning, so tweets were coded on the basis of the content available: text, punctuation, hashtags, and quote tweets. Some tweets from the 2019 sample had since been deleted and couldn't be verified for additional context that might have influenced their meaning.
Second, time constraints limited the number of criteria that could be coded, and the coding book itself couldn't be piloted on a smaller sample before full use.
Third, only one coder reviewed the tweets. Since interpretation of sentiment can vary by individual, inter-rater reliability couldn't be calculated to assess consistency across the classification categories. Future studies would benefit from additional coders to help maintain consistency.