The improvement directions of Recommendation System

Recommendation system is the basis of statistical speculation or statistical inference, or more generated-machine learning. This field is now developing rapidly, attracting many CS students to do research in. However, if we really want to get the consumers satisfied, apart from the aspect of SVM and neural network algorithms, this matter is heavily dependent on the following areas.

Satisfaction is a very vague concept. Maybe we can define it in this way-the a consumer can have an experience of excited feeling on upon catching sight of  what’s just in his subconsciousness. But this requirement is too high. It is possible for him to choose from the scope of which there is simply not the commodity that satisfies him. Such as Douban Radio, if he does not like to listen to classical music, whatever strong the recommendation algorithm is, he still can not be satisfied. In this case, no matter how well the recommender knows of him, it is difficult to give an answer.

Fortunately, there is recently a new trend among start-ups in Silicon Valley. They combine human recommendation and machine recommendation, which is also one key factor for Facebook and Twitter to be so charming. With the help of the deep developed API of Facebook and Twitter, those start-ups are reading our friend posts to enhance their training model for recommendation. This can have bad impacts also. If one have too many friends on Facebook or too many followers on Twitter, there can be some noise if a recommendation system has referred to it. This combination can be beneficial but can not solve all the problems.

The amount of data is growing fast this decade which can be a decade of fast user accumulation in the circumstance of Web 2.0. Statistics speculated that the most critical user data, which basically can not make bricks without straw in web 2.0. With the rapid development of the mass storage and concurrent systems, we can have a lot of data to be excavated. To sum up, the recommendation system is indeed much more practical than pure algorithms thus requiring a good mixture of the real demand and algorithm optimization.

Algorithms used in SNA

In last class we have learned the usage of shortest path in SNA(social network analysis), such as the calculation of “Closeness” or “Centrality”. To realize it, there are several famous algorithms, including “Dijkstra Algorithm”, “Floyd Algorithm” and “Prim Algorithm”. Following is a simple description of Dijkstra.

Let the node at which we are starting be called the initial node. Let the distance of node Y be the distance from the initial node to Y. Dijkstra algorithm will assign some initial distance values and will try to improve them step by step.

1. Assign to every node a tentative distance value: set it to zero for our initial node and to infinity for all other nodes.

2. Mark all nodes unvisited. Set the initial node as current. Create a set of the unvisited nodes called the unvisited set consisting of all the nodes.

3. For the current node, consider all of its unvisited neighbors and calculate their tentative distances. Compare the newly calculated tentative distance to the current assigned value and assign the smaller one.

4. When we are considering all of the neighbors of the current node, mark the current node as visited and remove it from the unvisited set. A visited node will never be checked again.

5. If the destination node has been marked visited (when planning a route between two specific nodes) or if the smallest tentative distance among the nodes in the unvisited set is infinity (when planning a complete traversal; occurs when there is no connection between the initial node and remaining unvisited nodes), then stop. The algorithm has finished.

6. Select the unvisited node that is marked with the smallest tentative distance, and set it as the new "current node" then go back to step 3.

It is also widely used in many other places. For example, in Google Maps, we often try to find the path to a certain building or some city, and the app will suggest us a way which is the shortest(shortest in distance or time). Using the same thinking, we get the shortest way to the next crossing or neighbor city, then repeat this process until finally get the whole path to destination.

Pathway from Hong Kong to Beijing by Google Map

Likely, in computer networks, routing strategies can help a router find the path to send packets to the next router, of which the widely used protocol-OSPF(Open Shortest Path First Interior Gateway Protocol) is just based on Dijkstra Algorithm.

About Sentiment Analysis

As an important part of Natural Language Processing, Sentiment Analysis, also known as orientation analysis, opinion extraction, opinion mining, emotional mining or subjective analysis is the process of analysis, management, induction and reasoning of emotional text with subjectivity.

Websites such as Taobao, Amazon, Douban are full of comments from the consumers on goods, books or movies. By analyzing the feelings most consumers convey, we can get the market response of a certain product. More intuitively, level of scores, for example 0.0~5.0 are used on the evaluation of a product. Higher scores are usually better evaluation. The movie “Dearest” by Peter Chan got a score of 8.6/10.0 which is the highest one of this year among Chinese movies, and we can know that this is a movie worth watching to some extent.

In May 2012, by analyzing Twitter users’ emotional trend to forecast stock trend, the world's first hedge fund based on social media-“Derwent Capital Markets” realized the rate of return 1.85% which is much higher than the average rate of 0.76% in this industry.

On Coursera, there is a course named “Natural Language Processing” by Stanford University. The 7^th class of this course is just about sentiment analysis. We can learn about a baseline algorithm on sentiment analysis. This algorithm including three major steps, namely tokenization(text extraction), feature extraction and classification using different classifiers(Naïve Bayes). Below is the training process using Naïve Bayes.