Publications

2023

1. Scalable Prompt Generation for Semi-supervised Learning with Language Models Zhou, Yuhang, Maharjan, Suraj, and Liu, Beiye In Findings of the Association for Computational Linguistics: EACL 2023 [To Appear], May 2023 [abstract]

   Prompt-based learning methods in semi-supervised learning (SSL) settings have been shown to be effective on multiple natural language understanding (NLU) datasets and tasks in the literature. However, manually designing multiple prompts and verbalizers requires domain knowledge and human effort, making it difficult and expensive to scale across different datasets. In this paper, we propose two methods to automatically design multiple prompts and integrate automatic verbalizer in SSL settings without sacrificing performance. The first method uses various demonstration examples with learnable continuous prompt tokens to create diverse prompt models. The second method uses a varying number of soft prompt tokens to encourage language models to learn different prompts. For the verbalizer, we use the prototypical verbalizer to replace the manual one. In summary, we obtained the best average accuracy of 73.2% (a relative improvement of 2.52% over even the previous state-of-the-art SSL method with manual prompts and verbalizers) in different few-shot learning settings

2020

1. C1 at SemEval-2020 Task 9: SentiMix: Sentiment Analysis for Code-Mixed Social Media Text Using Feature Engineering Advani, Laksh, Lu, Clement, and Maharjan, Suraj In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Dec 2020 [abstract] [link]

   In today’s interconnected and multilingual world, code-mixing of languages on social media is a common occurrence. While many Natural Language Processing (NLP) tasks like sentiment analysis are mature and well designed for monolingual text, techniques to apply these tasks to code-mixed text still warrant exploration. This paper describes our feature engineering approach to sentiment analysis in code-mixed social media text for SemEval-2020 Task 9: SentiMix. We tackle this problem by leveraging a set of hand-engineered lexical, sentiment, and metadata fea- tures to design a classifier that can disambiguate between “positive”, “negative” and “neutral” sentiment. With this model we are able to obtain a weighted F1 score of 0.65 for the “Hinglish” task and 0.63 for the “Spanglish” tasks.

2. Multiple social platforms reveal actionable signals for software vulnerability awareness: A study of GitHub, Twitter and Reddit Shrestha, Prasha, Sathanur, Arun, Maharjan, Suraj, Saldanha, Emily, Arendt, Dustin, and Volkova, Svitlana PLOS ONE, Mar 2020 [abstract] [link]

   The awareness about software vulnerabilities is crucial to ensure effective cybersecurity practices, the development of high-quality software, and, ultimately, national security. This awareness can be better understood by studying the spread, structure and evolution of software vulnerability discussions across online communities. This work is the first to evaluate and contrast how discussions about software vulnerabilities spread on three social platforms—Twitter, GitHub, and Reddit. Moreover, we measure how user-level e.g., bot or not, and content-level characteristics e.g., vulnerability severity, post subjectivity, targeted operating systems as well as social network topology influence the rate of vulnerability discussion spread. To lay the groundwork, we present a novel fundamental framework for measuring information spread in multiple social platforms that identifies spread mechanisms and observables, units of information, and groups of measurements. We then contrast topologies for three social networks and analyze the effect of the network structure on the way discussions about vulnerabilities spread. We measure the scale and speed of the discussion spread to understand how far and how wide they go, how many users participate, and the duration of their spread. To demonstrate the awareness of more impactful vulnerabilities, a subset of our analysis focuses on vulnerabilities targeted during recent major cyber-attacks and those exploited by advanced persistent threat groups. One of our major findings is that most discussions start on GitHub not only before Twitter and Reddit, but even before a vulnerability is officially published. The severity of a vulnerability contributes to how much it spreads, especially on Twitter. Highly severe vulnerabilities have significantly deeper, broader and more viral discussion threads. When analyzing vulnerabilities in software products we found that different flavors of Linux received the highest discussion volume. We also observe that Twitter discussions started by humans have larger size, breadth, depth, adoption rate, lifetime, and structural virality compared to those started by bots. On Reddit, discussion threads of positive posts are larger, wider, and deeper than negative or neutral posts. We also found that all three networks have high modularity that encourages spread. However, the spread on GitHub is different from other networks, because GitHub is more dense, has stronger community structure and assortativity that enhances information diffusion. We anticipate the results of our analysis to not only increase the understanding of software vulnerability awareness but also inform the existing and new analytical frameworks for simulating information spread e.g., disinformation across multiple social environments online.

2019

1. Book analysis and recommendation Payne, Holly Lynn, Bregman, Mark Fielding, Vargas, Bogart, Solorio, Thamar, Maharjan, Suraj, and Kar, Sudipta US Patent 10 503 829, Dec 2019 [abstract] [link]

   A method includes generating style values and experiential language tags (ELTs) for a plurality of books based on retrieved book content and reader reviews, respectively. The method further includes generating an ELT prediction model based on the style values and the ELTs. The ELT prediction model is configured to receive a set of style values for a new book and output a set of predicted ELTs for the new book, the set of predicted ELTs indicating predicted reader experiences with the new book. The method further includes receiving user-submitted book content from a remote user device, determining style values for the user-submitted book content, and determining a list of predicted ELTs for the user-submitted book content using the style values for the user-submitted book content and the ELT prediction model. Additionally, the method includes transmitting, to the user device, the list of predicted ELTs for the user-submitted book content.

2. Learning from Dynamic User Interaction Graphs to Forecast Diverse Social Behavior Shrestha, Prasha, Maharjan, Suraj, Arendt, Dustin, and Volkova, Svitlana In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM), Nov 2019 [abstract] [pdf]

   Most of the existing graph analytics for understanding social be- havior focuses on learning from static rather than dynamic graphs using hand-crafted network features or recently emerged graph embeddings learned independently from a downstream predictive task, and solving predictive (e.g., link prediction) rather than fore- casting tasks directly. To address these limitations, we propose (1) a novel task – forecasting user interactions over dynamic social graphs, and (2) a novel deep learning, multi-task, node-aware at- tention model that focuses on forecasting social interactions, going beyond recently emerged approaches for learning dynamic graph embeddings. Our model relies on graph convolutions and recurrent layers to forecast future social behavior and interaction patterns in dynamic social graphs. We evaluate our model on the ability to forecast the number of retweets and mentions of a specific news source on Twitter (focusing on deceptive and credible news sources) with R2 of 0.79 for retweets and 0.81 for mentions. An additional evaluation includes model forecasts of user-repository interactions on GitHub and comments to a specific video on YouTube with a mean absolute error close to 2% and R2 exceeding 0.69. Our results demonstrate that learning from connectivity information over time in combination with node embeddings yields better forecasting results than when we incorporate the state-of-the-art graph em- beddings e.g., Node2Vec and DeepWalk into our model. Finally, we perform in-depth analyses to examine factors that influence model performance across tasks and different graph types e.g., the influence of training and forecasting windows as well as graph topological properties.

3. Jointly Learning Author and Annotated Character N-gram Embeddings: A Case Study in Literary Text Maharjan, Suraj, Mave, Deepthi, Shrestha, Prasha, Montes, Manuel, González, Fabio A., and Solorio, Thamar In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP), Sep 2019 [abstract] [pdf]

   An author’s way of presenting a story through his/her writing style has a great impact on whether the story will be liked by readers or not. In this paper, we learn representations for authors of literary texts together with representations for character n-grams annotated with their functional roles. We train a neural character n-gram based language model using an external corpus of literary texts and transfer learned representations for use in downstream tasks. We show that augmenting the knowledge from external works of authors produces results competitive with other style-based methods for book likability prediction, genre classification, and authorship attribution.

2018

1. Language Identification and Analysis of Code-Switched Social Media Text Mave, Deepthi, Maharjan, Suraj, and Solorio, Thamar In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching (CALCS), Jul 2018 [abstract] [pdf]

   In this paper, we detail our work on comparing different word-level language identification systems for code-switched Hindi-English data and a standard Spanish-English dataset. In this regard, we build a new code-switched dataset for Hindi-English. To understand the code-switching patterns in these language pairs, we investigate different code-switching metrics. We find that the CRF model outperforms the neural network based models by a margin of 2-5 percentage points for Spanish-English and 3-5 percentage points for Hindi-English.

2. Letting Emotions Flow: Success Prediction by Modeling the Flow of Emotions in Books Maharjan, Suraj, Kar, Sudipta, Montes, Manuel, González, Fabio A., and Solorio, Thamar In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Jun 2018 [abstract] [link] [pdf] [poster] [code]

   Books have the power to make us feel happiness, sadness, pain, surprise, or sorrow. An author’s dexterity in the use of these emotions captivates readers and makes it difficult for them to put the book down. In this paper, we model the flow of emotions over a book using recurrent neural networks and quantify its usefulness in predicting success in books. We obtained the best weighted F1-score of 69% for predicting books’ success in a multitask setting (simultaneously predicting success and genre of books).

3. A Genre-Aware Attention Model to Improve the Likability Prediction of Books Maharjan, Suraj, Montes, Manuel, González, Fabio A., and Solorio, Thamar In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct 2018 [abstract] [pdf]

   Likability prediction of books has many uses. Readers, writers, as well as the publishing industry, can all benefit from automatic book likability prediction systems. In order to make reliable decisions, these systems need to assimilate information from different aspects of a book in a sensible way. We propose a novel multimodal neural architecture that incorporates genre supervision to assign weights to individual feature types. Our proposed method is capable of dynamically tailoring weights given to feature types based on the characteristics of each book. Our architecture achieves competitive results and even outperforms state-of-the-art for this task.

4. Folksonomication: Predicting Tags for Movies from Plot Synopses using Emotion Flow Encoded Neural Network Kar, Sudipta, Maharjan, Suraj, and Solorio, Thamar In Proceedings of the 27th International Conference on Computational Linguistics (COLING), Aug 2018 [abstract] [pdf]

   Folksonomy of movies covers a wide range of heterogeneous information about movies, like the genre, plot structure, visual experiences, soundtracks, metadata, and emotional experiences from watching a movie. Being able to automatically generate or predict tags for movies can help recommendation engines improve retrieval of similar movies, and help viewers know what to expect from a movie in advance. In this work, we explore the problem of creating tags for movies from plot synopses. We propose a novel neural network model that merges information from synopses and emotion flows throughout the plots to predict a set of tags for movies. We compare our system with multiple baselines and found that the addition of emotion flows boosts the performance of the network by learning \mbox≈18% more tags than a traditional machine learning system.

5. MPST: A Corpus of Movie Plot Synopses with Tags Kar, Sudipta, Maharjan, Suraj, López-Monroy, A. Pastor, and Solorio, Thamar In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), May 2018 [abstract] [pdf] [slides]

   Social tagging of movies reveals a wide range of heterogeneous information about movies, like the genre, plot structure, soundtracks, metadata, visual and emotional experiences. Such information can be valuable in building automatic systems to create tags for movies. Automatic tagging systems can help recommendation engines to improve the retrieval of similar movies as well as help viewers to know what to expect from a movie in advance. In this paper, we set out to the task of collecting a corpus of movie plot synopses and tags. We describe a methodology that enabled us to build a fine-grained set of around 70 tags exposing heterogeneous characteristics of movie plots and the multi-label associations of these tags with some 14K movie plot synopses. We investigate how these tags correlate with movies and the flow of emotions throughout different types of movies. Finally, we use this corpus to explore the feasibility of inferring tags from plot synopses. We expect the corpus will be useful in other tasks where analysis of narratives is relevant.

6. Towards Anticipatory Analytics: Forecasting Instability Across Countries from Dynamic Knowledge Graphs Maharjan, Suraj, Shrestha, Prasha, Porterfield, Katherine, Arendt, Dustin, and Volkova, Svitlana In Proceedings of the 5th Pacific Northwest Regional NLP Workshop (NW-NLP), Apr 2018 [abstract] [pdf] [poster]

   Protests, civil unrest, and instability around the world not only cause physical damages but also human casualties. It would be possible to mitigate these losses if there were a way to forecast such events happening world-wide. With the plethora of information coming from a variety of news sources, a system can be designed to alert about such events in advance. The ability to forecast protests, fights, and assaults can help governments to make plans to safeguard citizens from potential casualties, and monitor essential supplies.

2017

1. A Multi-task Approach for Named Entity Recognition in Social Media Data Aguilar, Gustavo, Maharjan, Suraj, López-Monroy, Adrian Pastor, and Solorio, Thamar In Proceedings of the 3rd Workshop on Noisy User-generated Text (W-NUT), Sep 2017 [abstract] [pdf]

   Named Entity Recognition for social media data is challenging because of its inherent noisiness. In addition to improper grammatical structures, it contains spelling inconsistencies and numerous informal abbreviations. We propose a novel multi-task approach by employing a more general secondary task of Named Entity (NE) segmentation together with the primary task of fine-grained NE categorization. The multi-task neural network architecture learns higher order feature representations from word and character sequences along with basic Part-of-Speech tags and gazetteer information. This neural network acts as a feature extractor to feed a Conditional Random Fields classifier. We were able to obtain the first position in the 3rd Workshop on Noisy User-generated Text (WNUT-2017) with a 41.86% entity F1-score and a 40.24% surface F1-score.

2. Detecting Nastiness in Social Media Safi Samghabadi, Niloofar, Maharjan, Suraj, Sprague, Alan, Diaz-Sprague, Raquel, and Solorio, Thamar In Proceedings of the First Workshop on Abusive Language Online (ALW), Aug 2017 [abstract] [pdf] [poster]

   Although social media has made it easy for people to connect on a virtually unlimited basis, it has also opened doors to people who misuse it to undermine, harass, humiliate, threaten and bully others. There is a lack of adequate resources to detect and hinder its occurrence. In this paper, we present our initial NLP approach to detect invective posts as a first step to eventually detect and deter cyberbullying. We crawl data containing profanities and then determine whether or not it contains invective. Annotations on this data are improved iteratively by in-lab annotations and crowdsourcing. We pursue different NLP approaches containing various typical and some newer techniques to distinguish the use of swear words in a neutral way from those instances in which they are used in an insulting way. We also show that this model not only works for our data set, but also can be successfully applied to different data sets.

3. RiTUAL-UH at SemEval-2017 Task 5: Sentiment Analysis on Financial Data Using Neural Networks Kar, Sudipta, Maharjan, Suraj, and Solorio, Thamar In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval), Aug 2017 [abstract] [pdf] [poster] [code]

   In this paper, we present our systems for the “SemEval-2017 Task-5 on Fine-Grained Sentiment Analysis on Financial Microblogs and News”. In our system, we combined hand-engineered lexical, sentiment and metadata features, the representations learned from Convolutional Neural Networks (CNN) and Bidirectional Gated Recurrent Unit (Bi-GRU) with Attention model applied on top. With this architecture we obtained weighted cosine similarity scores of 0.72 and 0.74 for subtask-1 and subtask-2, respectively. Using the official scoring system, our system ranked the second place for subtask-2 and eighth place for the subtask-1. It ranked first for both of the subtasks by the scores achieved by an alternate scoring system.

4. A Multi-task Approach to Predict Likability of Books Maharjan, Suraj, Arevalo, John, Montes, Manuel, González, Fabio A., and Solorio, Thamar In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Apr 2017 [abstract] [pdf] [slides]

   We investigate the value of feature engineering and neural network models for predicting successful writing. Similar to previous work, we treat this as a binary classification task and explore new strategies to automatically learn representations from book contents. We evaluate our feature set on two different corpora created from Project Gutenberg books. The first presents a novel approach for generating the gold standard labels for the task and the other is based on prior research. Using a combination of hand-crafted and recurrent neural network learned representations in a dual learning setting, we obtain the best performance of 73.50% weighted F1-score.

2016

1. CogALex-V Shared Task: GHHH - Detecting Semantic Relations via Word Embeddings Attia, Mohammed, Maharjan, Suraj, Samih, Younes, Kallmeyer, Laura, and Solorio, Thamar In Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex), Dec 2016 [abstract] [pdf] [poster]

   This paper describes our system submission to the CogALex-2016 Shared Task on Corpus-Based Identification of Semantic Relations. Our system won first place for Task-1 and second place for Task-2. The evaluation results of our system on the test set is 88.1% (79.0% for TRUE only) f-measure for Task-1 on detecting semantic similarity, and 76.0% (42.3% when excluding RANDOM) for Task-2 on identifying finer-grained semantic relations. In our experiments, we try word analogy, linear regression, and multi-task Convolutional Neural Networks (CNNs) with word embeddings from publicly available word vectors. We found that linear regression performs better in the binary classification (Task-1), while CNNs have better performance in the multi-class semantic classification (Task-2). We assume that word analogy is more suited for deterministic answers rather than handling the ambiguity of one-to-many and many-to-many relationships. We also show that classifier performance could benefit from balancing the distribution of labels in the training data.

2. Multilingual Code-switching Identification via LSTM Recurrent Neural Networks Samih, Younes, Maharjan, Suraj, Attia, Mohammed, Kallmeyer, Laura, and Solorio, Thamar In Proceedings of the Second Workshop on Computational Approaches to Code Switching (CALCS), Nov 2016 [abstract] [pdf] [slides] [code]

   This paper describes the HHU-UH-G system submitted to the EMNLP 2016 Second Workshop on Computational Approaches to Code Switching. Our system ranked first place for Arabic (MSA-Egyptian) with an F1-score of 0.83 and second place for Spanish-English with an F1-score of 0.90. The HHU-UHG system introduces a novel unified neural network architecture for language identification in code-switched tweets for both SpanishEnglish and MSA-Egyptian dialect. The system makes use of word and character level representations to identify code-switching. For the MSA-Egyptian dialect the system does not rely on any kind of language-specific knowledge or linguistic resources such as, Part Of Speech (POS) taggers, morphological analyzers, gazetteers or word lists to obtain state-ofthe-art performance.

2015

1. Using Wide Range of Features for Author profiling. Maharjan, Suraj, and Solorio, Thamar In Notebook for PAN at CLEF, Sep 2015 [abstract] [pdf]

   Predicting an author’s age, gender and personality traits by analyzing his/her documents is important in forensics, marketing and resolving authorship disputes. Our system combines different styles, lexicons, topics, familial tokens and different categories of character n-grams as features to build a logistic regression model for four different languages: English, Spanish, Italian and Dutch. With this model, we obtained global ranking scores of 0.6623, 0.6547, 0.7411, 0.7662 for English, Spanish, Italian and Dutch languages respectively.

2. Developing Language-tagged Corpora for Code-switching Tweets Maharjan, Suraj, Blair, Elizabeth, Bethard, Steven, and Solorio, Thamar In Proceedings of The 9th Linguistic Annotation Workshop (LAW), Jun 2015 [abstract] [pdf] [poster]

   Code-switching, where a speaker switches between languages mid-utterance, is frequently used by multilingual populations worldwide. Despite its prevalence, limited effort has been devoted to develop computational approaches or even basic linguistic resources to support research into the processing of such mixed language data. We present a user-centric approach to collecting code-switched utterances from social media posts, and develop language universal guidelines for the annotation of codeswitched data. We also present results for several baseline language identification models on our corpora and demonstrate that language identification in code-switched text is a difficult task that calls for deeper investigation.

2014

1. Using String Information for Malware Family Identification Shrestha, Prasha, Maharjan, Suraj, Rosa, Gabriela Ramírez, Sprague, Alan, Solorio, Thamar, and Warner, Gary In Advances in Artificial Intelligence – Ibero-American Conference on Artificial Intelligence (IBERAMIA), Nov 2014 [abstract] [pdf]

   Classifying malware into correct families is an important task for anti-virus vendors. Currently, only some of them will recognize a particular malware. Even when they do, they either classify them into different families or use a generic family name, which does not provide much information. Our method for malware family identification is based on the observation that closely related malware have heavy overlap of strings. We first created two kinds of prototypes from printable strings in the malware: one using term frequency–inverse document frequency (tf-idf) and the other using the prominent strings extracted from the vocabulary. We then used these prototypes for classification. We achieved an accuracy of 91.02 % by considering the entire vocabulary and an accuracy of 80.52 % by considering 20 prominent strings for each malware family. Our accuracy is high enough for our system to be used to classify even those malware that can confuse the anti-virus vendors.

2. A Straightforward Author Profiling Approach in MapReduce Maharjan, Suraj, Shrestha, Prasha, Solorio, Thamar, and Hasan, Ragib In Advances in Artificial Intelligence – Ibero-American Conference on Artificial Intelligence (IBERAMIA), Nov 2014 [abstract] [pdf]

   Most natural language processing tasks deal with large amounts of data, which takes a lot of time to process. For better results, a larger dataset and a good set of features are very helpful. But larger volumes of text and high dimensionality of features will mean slower performance. Thus, natural language processing and distributed computing are a good match. In the PAN 2013 competition, the test runtimes for author profiling range from several minutes to several days. Most author profiling systems available now are either inaccurate or slow or both. Our system, written entirely in MapReduce, employs nearly 3 million features and still manages to finish the task in a fraction of time than state-of-the-art systems and with better accuracy. Our system demonstrates that when we deal with a huge amount of data and/or a large number of features, using distributed systems makes perfect sense.

3. Overview for the First Shared Task on Language Identification in Code-Switched Data Solorio, Thamar, Blair, Elizabeth, Maharjan, Suraj, Bethard, Steven, Diab, Mona, Ghoneim, Mahmoud, Hawwari, Abdelati, AlGhamdi, Fahad, Hirschberg, Julia, Chang, Alison, and Fung, Pascale In Proceedings of the First Workshop on Computational Approaches to Code Switching, Oct 2014 [abstract] [pdf]

   We present an overview of the first shared task on language identification on codeswitched data. The shared task included code-switched data from four language pairs: Modern Standard Arabic-Dialectal Arabic (MSA-DA), Mandarin-English (MAN-EN), Nepali-English (NEP-EN), and Spanish-English (SPA-EN). A total of seven teams participated in the task and submitted 42 system runs. The evaluation showed that language identification at the token level is more difficult when the languages present are closely related, as in the case of MSA-DA, where the prediction performance was the lowest among all language pairs. In contrast, the language pairs with the higest F-measure where SPA-EN and NEP-EN. The task made evident that language identification in code-switched data is still far from solved and warrants further research.

4. A Simple Approach to Author Profiling in MapReduce. Maharjan, Suraj, Shrestha, Prasha, and Solorio, Thamar In Notebook for PAN at CLEF, Sep 2014 [abstract] [pdf] [poster] [slides]

   Author profiling, being an important problem in forensics, security, marketing, and literary research, needs to be accurate. With massive amounts of online text readily available on which we might need to perform author profiling, building a fast system is as important as building an accurate system, but this can be challenging. However, the use of distributive computing techniques like MapReduce can significantly lower processing time by distributing tasks across multiple machines. Our system uses MapReduce programming paradigm for most parts of the training process, which makes our system fast. Our system uses word n-grams including stopwords, punctuations and emoticons as features and TF-IDF (term frequency inverse document frequency) as the weighing scheme. These are fed to the logistic regression classifier that predicts the age and gender of the authors. We were able to obtain a competitive accuracy in most categories and even obtained winning accuracy for two of the categories each in both test corpus 1 and test corpus 2.

5. Machine Translation Evaluation Metric for Text Alignment. Shrestha, Prasha, Maharjan, Suraj, and Solorio, Thamar In Notebook for PAN at CLEF, Sep 2014 [abstract] [pdf] [poster]

   As plagiarisers become cleverer, plagiarism detection becomes harder. Plagiarisers will find new ways to obfuscate the plagiarized passages so that humans and automatic plagiarism detectors are not able to point them out. So, a plagiarism detection system needs to be robust enough to detect plagiarism, no matter what obfuscation techniques have been applied. Our system attempts to do the same by combining two different methods, one stricter to catch mildly obfuscated passages and one more lenient to catch difficult ones. We use a machine translation evaluation metric and n-gram matching to detect overlaps between the source and suspicious documents. On the PAN’14 corpus, which contains data with various types of obfuscation to mimic human plagiarisers, we obtained plagdet scores of 0.84404 and 0.86806 on the two datasets.

2011

1. Product recommendations using linear predictive modeling Banjade, Rajendra, and Maharjan, Suraj In 2011 Second Asian Himalayas International Conference on Internet (AH-ICI), Nov 2011 [abstract] [pdf]

   Recommendation systems apply statistical and knowledge discovery techniques to the problem of making product recommendations and they are achieving widespread success in E-Commerce these days. A successful recommendation system fulfils several purposes and the choice of the methodology significantly influences the quality of recommendations and other aspects including scalability. As the volume of data in the e-commerce is growing massively, the system should also be able to address the need to provide the recommendations either by in-memory calculations or offline calculations, both demanding the high performance. For a large number of customers and products, the linear regression with a proper model selection can provide significantly better results and performance. Recommendations engines are increasingly becoming a popular choice for solving the problem of content discovery enabling the user to find personally relevant content that they might not have known was available. In this paper, we consider linear regression technique for analyzing large-scale dataset for the purpose of useful recommendations to e-commerce customers by offline calculations of model results.