Information Retrieval on Weimao Ke

Information Theory

Wed, 27 Apr 2016 00:00:00 +0000

What is information? How can we quantify the amount of information? These are fundamental and challenging questions. They are fundamental because a broad spectrum of problems we face are centered on the notion of \textbf{information} and how it can be measured in practical applications. Yet there is hardly any agreement on what it is and how it should be treated. From \textit{Shannon Entropy} to \textit{Landauer’s principle} and Wheeler’s ``It from Bit,’’ there are perspectives in which information can be viewed and measured.

A better understanding of these questions through the lenses of related theories and discoveries will provide insight into new models for improved information representation, dissemination, and analytics. Rooted in Shannon’s entropic framework of information, my research on the Least Information Theory (LIT) and the Discounted LIT of Entropy (DLITE or pronounced \textit{delight}) attempts to address very practical problems in information processing by creating a new theory with several desirable properties. It was a risky research project I initiated as a junior faculty member which, fortunately, has yielded extraordinary fruit on both the theoretical and practical levels.

Information theory provides the foundation and guiding principles for research in search and analytics. In the formulation of LIT and DLITE as potential quantities of information, we discovered multiple principle properties desirable in information retrieval and text mining tasks.

For example, classic Inverse Document Frequency (IDF) scoring computes the amount of Kullback-Leibler (KL) divergence in a term (word) as its term weight. It is often assumed that these weights can be added up as a distance metric and can be compared on the same scale when their theoretical characteristics suggest otherwise. The development of DLITE addresses these issues because of its distinctive properties:

DLITE is a \textit{volumetric} and its cube root a \textit{distance metric} that satisfies properties such as triangular inequality. It is fitting to use it in classic scoring functions where the sum of such values (scores) are meaningful.
DLITE is normalized and bounded. As such, it mitigates issues of dominant terms in the scoring/ranking function, e.g. a rare term that renders other query terms useless.
As the DLITE of an ensemble (the entire system) can be computed as the weighted sum of its sub-systems, it offers researchers the capacity to dissect and reconstruct a system in different ways without altering the final scoring.

These properties have enabled the creation of a new IR ranking system with superior performances, which I will elaborate in the next section.

Library Chat Analytics

Wed, 27 Apr 2016 00:00:00 +0000

Recent advances in machine learning (ML) and large language models (LLMs) are offering exciting opportunities for libraries filled with precious data. This potential shines brightly in areas like chat-based assistance. Yet, there are some roadblocks - concerns about data privacy, the risk of models giving inaccurate information (hallucination), and the expenses related to tailoring and assessing these tools specifically for libraries.

Information Retrieval, often linked to the term ‘Reference Retrieval’, is deeply tied to library studies and services. This approach can help navigate challenges faced by LLMs. It gives us a solid framework to analyze data-driven responses. In this process, librarians also play a pivotal role and their insights and feedback are vital in ensuring that this blend of traditional library methods and cutting-edge AI truly benefits library administrators and users.

Search & Retrieval

Wed, 27 Apr 2016 00:00:00 +0000

A thread of my research is on big data (due to its volume and variety) and how distributed systems with machine learning (ML) can adapt and scale. With the objective to develop a large-scale decentralized search engine, we draw on inspirations from research in complex networks and multi-agent systems. We showed that efficient decentralized searches are possible in a distributed network of agents only when the network structure is optimized. According to a phenomenon we refer to as the \textit{Clustering Paradox}, there is a specific level of network clustering, i.e. grouping of neighbor agents with an ideal proportion of similar vs. dissimilar contents, where searches can be most efficient. Either over-clustering or under-clustering will lead to degraded search performances. Finding the ideal level is key to the success of agent collaboration for decentralized information seeking.

Fortunately such an ideal network structure can be achieved without a global knowledge or hard engineering in a top-down manner. Rather, agents can interconnect locally and organically to construct this global structure from bottom up. Using a connectivity probability function that favors neighbors with similar contents but allows small chances to connect with those that are different, the network of these agent communities will emerge with the following two types of links (or ties): 1) strong ties that define local communities and serve as anchors (labels) to direct searches toward their relevant target, and 2) weak ties that connect different communities together and serve as ``hubs’’ for searches to jump from one community to another.

Studying the clustering paradox and scalability of search in highly distributed environments

Tue, 01 Jan 2013 00:00:00 +0000

Supplementary notes can be added here, including code and math.