7 3 09/05 - 09/09 [Virtual Cloudera] Import the original small collection into HBase Done Anusha, Ke 8 4 09/12 - 09/16 [Virtual Cloudera] Create a Solr instance for the original small collection Done Anusha, Ke 9M 4 09/12 - 09/16 [Virtual Cloudera] Use Lily Indexer to map columns in HBase with fields in Solr and make a basic query test (Live Mode) Done Liuqing 10 4 09/12 - 09/16 [Hadoop Cluster] Import the original small collection into HBase Done Liuqing 11 4 09/12 - 09/16 [Hadoop Cluster] Create a Solr instance for the original small collection Done Liuqing 12D 4, 5 09/12 - 09/23 Share the previous schema file with relevant teams for modification Done All 13D 5 09/19 - 09/23 [Hadoop Cluster] Use Lily Indexer to map columns in HBase with fields in Solr for FE team future test (Live Mode) Done Liuqing 14 6 09/26 - 09/30 Update the stoplist.txt and profanity.txt Done All 15 6 09/26 - 09/30 Divide our team into two subgroups for the NRT and recommender tasks Done All 16M 6 09/26 - 09/30 [Virtual Cloudera] Use Lily Indexer to accomplish the offline indexing job (Batch Mode) Done All 17M 7 10/03 - 10/07 [Virtual Cloudera] Use Lily Indexer to accomplish the incremental indexing job (NRT) Done Liuqing, Ke 18 6, 7, 8 09/26 - 10/14 Learn the knowledge of faceted search and recommender Done Anusha, Ye
19M 8 10/10 - 10/14 [Virtual Cloudera] Modify the schema file to make a facet search query Done Ke 20D 8 10/10 - 10/14 [Hadoop Cluster] Use Lily Indexer to accomplish the incremental indexing (NRT) Postponed (No. 29) Liuqing 21M 9 10/17 - 10/21 Design and Implement a basic recommender Done Anusha, Ye 22 10 10/24 - 10/28 Learn the knowledge of custom ranking function Done Liuqing 23 10 10/24 - 10/28 Further work on facet search Done Ke 24D 11 10/31 - 11/04 Create the schema.xml based on the HBase structure Done All 25 12 11/07 - 11/11 Further work on recommender Done Anusha, Ye 26 12 11/07 - 11/11 Build custom ranking handler Done Liuqing, Ke 27M 12, 13 11/07 - 11/18 Index the big dataset (raw tweets) Done All 28 13 11/14 - 11/18 Deploy the MoreLikeThis handler and custom ranking component Done All 29 14 11/21 - 11/25 [Hadoop Cluster] Use Lily Indexer to accomplish the incremental indexing (NRT) Done Liuqing 30 14 11/21 - 11/25 Learn about SolrCloud Done Anusha, Ye, Ke 31M 15 11/28 - End Index the big dataset (raw tweets and processed data) Done Liuqing M – Milestones D – Deliverables User Manual In this section, we describe how a user without comprehensive background on Solr could utilize Solr for query and search. How to use Solr for indexing and querying is not intuitive. The purpose of this user manual is to provide more intuitive examples for users to work with Solr.
Figure 26 Generate the Index Files with Live Mode in Virtual Cloudera Note: If you get a Java heap space error, adjust the values of the mapreduce.map.java.opts and mapreduce.reduce.java.opts in the file /etc/hadoop/conf/mapred-site.xml. Figure 27 Fix the Java Heap Error in Virtual Cloudera Use Lily Indexer for Data Indexing (Batch Mode) The live mode of Lily Indexer is simple and efficient, because it makes full use of the memory. However, it cannot deal with big data. The batch mode of Lily indexer is helpful for indexing a large dataset. Create a Lily Indexer configuration file. This step is the same as the one in live mode. Run HBaseMapReduceIndexer tool. We index the HBase table using MapReduce in batch mode. The following command helps us to output the indexes into HDFS. Key Command(s) hadoop --config /etc/hadoop/conf jar /usr/lib/hbase-solr/tools/hbase-indexer-mr-*-job.jar /etc/hbase/conf/hbase-site.xml -D 'mapred.child.java.opts=-Xmx500m' [LOCAL_DIR]/morphline-hbase-mapper.xml --zk-host 127.0.0.1/solr --log4j [LOCAL_DIR]/log4j.properties --collection [COLLECTION_NAME] --verbose --output-dir hdfs://quickstart.cloudera/user/cloudera/cs5604f16-small-index --overwrite-output-dir 1 Figure 28 Generate the Index Files with Batch Mode in Virtual Cloudera the files in HDFS. If the MapReduce job goes successfully, then multiple indexing files have already been generated in HDFS. Key Command(s) hadoop fs -ls /solr/cs5604f16-small/core_node1/data/index Figure 29 the Index Files in HDFS in Virtual Cloudera
doc-type doc-type type of the document tweet / webpage tweet-topic label-list 1. labels generated by LDA model 2. extract the top two labels from each topic Signed,students; today,register; april,thanks; community,little probability-list each value presents the probability of the tweet belongs to a certain topic 0.29112; 0.01820; 0.12435; 0.02572; 0.54058 tweet-cluster cluster-label label of the tweet’s cluster NAACP stories doc-probability the probability of the doc in the cluster 0.55167194