AI model developed by LLYC to measure reputation outperforms competitors in accuracy, UCM says

The Complutense University of Madrid has certified that the AI model developed by LLYC to measure reputational polarity in social conversation is up to 20 points more accurate than the main market benchmarks (Amazon, Azure, Google and IBM). Specifically, it yields a level of 0.66 compared to a range of between 0.42 and 0.53 for the rival tools tested. The model is trained on more than 130,000 manually labeled messages.

“The results of the audit by the Complutense University of Madrid are an endorsement of the work that LLYC has done in recent years in terms of innovation. They represent a clear improvement on what has been available in the market up to now. Undoubtedly, achieving the most effective model for measuring reputation in social networks will be very useful for our clients,” says Daniel Fernández Trejo, Managing Director of Deep Learning at the firm.

This is how the project has developed

LLYC began analyzing the digital social conversation in depth in 2010. Since then, the firm realized that the sentiment metrics that social listening platforms (SLPs) offered (and still offer), are not very useful for assessing the evolution of a brand or company’s reputation. “I am very sad about Tina Turner’s passing” is a negative sentiment message that nevertheless reflects a positive reputational polarity of the author towards Tina Turner. The levels of error in measuring reputational damage or enhancement to a customer issue were such that to obtain a reliable metric, human analysis was necessary. Human analysis is more accurate, but, it does not scale (they cannot deal with large volumes of data), is slower and more expensive. Even so, it was the one used in methodologies and measurement tools developed by LLYC at the time, such as, for example, the MRO.

In 2021, three circumstances converge to open a window of opportunity for LLYC to drastically improve its reputation measurement and analysis capabilities. On the one hand, the Technology Area (now Deep Learning) begins to consolidate and provides the firm with technological capabilities that it did not have before. On the other hand, the state of the art in natural language processing (NLP) has matured a series of technologies (the Transformers) that have meant a disruptive leap in the capacity of machines to understand human language. Finally, thanks to the use of tools such as MRO over the last 10 years, LLYC has a database of more than 120,000 messages carefully classified by communication experts according to the reputational polarity they express. Could the current technology (Transformers), managed by a new in-house expertise (Technology Area) be trained with the accumulated knowledge (the 120,000 messages) to be able to measure reputation automatically and accurately?

To answer the above question, an innovation project codenamed Gea was launched to train a state-of-the-art AI with LLYC’s knowledge of reputation polarity. The information coming from the MRO was a good basis, although it was discovered that it was not enough to exploit all the possibilities offered by the technology: more data was needed. Thus, a new stage of human analysis was designed, involving more than 300 professionals from, at the time, Deep Digital, with the purpose of classifying 130,000 additional messages with which to better train Project Gea’s AI. The work has paid off. Now the Complutense University of Madrid certifies that the proprietary AI model developed by LLYC measures reputational polarity more accurately than other tools on the market.