Diesner and Mishra publish paper on NER tool for social media research

Twitter logo
Jana Diesner
Jana Diesner, Associate Professor and PhD Program Director
Shubhanshu Mishra
Shubhanshu Mishra

The identification of proper names of people, organizations, and locations from raw texts, referred to as Named Entity Recognition (NER), can be highly accurate when researchers use NER tools on a large collection of text with proper syntax. However, using existing NER tools for analyzing social media text can lead to poor identification of named entities. In particular, Twitter text frequently includes inconsistent capitalization, spelling errors, and shortened versions of words.

TwitterNER, an open-source tool developed by doctoral student Shubhanshu Mishra, who is supervised by Assistant Professor Jana Diesner, can help researchers interested in performing NER on social media text. TwitterNER has recently been shown (in an independent evaluation by Humangeo) to perform better in terms of precision than some other publicly available systems for entity types of person, location, and organization, which are often of most interest to researchers.

"Our system relies on a combination of hand-engineered features," explained Mishra. "It follows the paradigm of transductive semi-supervised learning where all the labeled and unlabeled data is utilized to make predictions about the unlabeled data."

The original implementation of TwitterNER was created for the shared-task session at the 2016 Conference on Computational Linguistics (COLING) Workshop on "Noisy User-generated Text" (W-NUT). Workshop participants were asked to build an NER system for Twitter data, which was evaluated using a common test dataset. TwitterNER had a high level of precision among the various systems.

Diesner and Mishra then improved their approach and shared it with W-NUT by submitting the paper, "Semi-supervised Named Entity Recognition in noisy-text."

"Our original submission ranked seventh in the task, but our final improved version surpassed the second-best performing system on the concluded task," said Mishra. "The winning system was based on deep learning, but its implementation is not publicly available."

Mishra has an integrated MS and BS in mathematics and computing from the Indian Institute of Technology Kharagpur. He is interested in the analysis of information generation in social networks such as those in scholarly data and social media websites. His prior projects have included systems for user sentiment profiling, active learning using human-in-the-loop design pattern, and novelty profiling in scholarly data.

Diesner is an expert in human-centered computing, network science, natural language processing, and machine learning. Recognition for her research expertise include appointments as CIO Scholar for Information Research & Technology at Illinois (2018), faculty fellow at the National Center for Supercomputing Applications (NCSA) at Illinois (2015), and as a research fellow in the Dori J. Maynard Senior Research Fellows program through The Center for Investigative Reporting and The Robert C. Maynard Institute for Journalism Education (2016). She holds a PhD from the Computation, Organizations and Society (COS) program at Carnegie Mellon University's School of Computer Science.

Updated on
Backto the news archive

Related News

Barbosa and Wang receive Facebook grant to design privacy controls for ad targeting

iSchool PhD student Natã Barbosa and his advisor Associate Professor Yang Wang have received a $65,053 grant from Facebook for their project, "In-Situ Privacy Controls of Profiling and Ad-Targeting." The goal of the project is to design a privacy control framework that makes profiling and ad-targeting more transparent to ordinary Internet users.

Yang Wang

Anderson selected as 2019-2021 iSchool research fellow

Theresa Dirndorfer Anderson has been selected by the iSchool faculty as a research fellow for the 2019-2021 academic years. Research fellows are chosen because their work is relevant to the interests of the School's faculty and students. During the period of their appointments, fellows give at least one public lecture.

Theresa Anderson

Student award recipients announced

Each year, the School recognizes a group of outstanding students for their achievement in academics as well as a number of attributes that contribute to professional success. Congratulations to this year's honorees!

Diesner joins Science Advances editorial board

Associate Professor and PhD Program Director Jana Diesner is a new associate editor on the editorial board of Science Advances, the open access multidisciplinary journal of the American Association for the Advancement of Science (AAAS). The journal supports the AAAS mission by extending the capacity of Science magazine to identify and promote significant advances in science and engineering across a wide range of areas. Science Advances editors not only have stellar reputations in their disciplines but also have acknowledged breadth in recognizing and promoting interdisciplinary collaborations. Diesner brings to this role her expertise in computational social science, human-centered data science, network analysis, natural language processing, machine learning, and responsible computing.

Assistant Professor Jana Diesner

La Barre recognized for diversity work

Associate Professor Kathryn La Barre received an Honorable Mention in the category of Outstanding Faculty/Staff at the 8th annual Diversity and Social Justice Education Awards. The awards recognize undergraduates, graduate students, faculty, staff, and student organizations "that have sought to address marginalization, oppression, and/or privilege in their communities." La Barre serves as chair of the iSchool's Diversity Committee.

Kathryn La Barre