21. Implement a custom ranking function for Solr collection on the Cluster 3/18 - 3/30 Done Andrej/Shivam 22. Configure Morphline mapping and transformation of HBase data fields into Solr document fields 3/19 - 3/25 Done Long/Tingting 23. Configure Extended DisMax Query Parser for boosting queries 3/22 - 3/28 Done Shivam 24. Test Solr schema and configurations on the 6 small collections on the cluster 3/26 - 4/2 Done Tingting/Long 25. Run Multiple Linear Regression using Apache Math Commons Java Library 3/29 - 4/4 Done Shivam 26. Adjust Solr field names and data representations given the feedbacks from other teams 4/3 - 4/11 Done Tingting/Long 27. Enable Velocity UI and configure custom SearchComponent in the cluster 4/8 - 4/12 Done Andrej 28. Batch index data input for 12 collections from all the teams in the cluster 4/12 - 4/20 Done Long/Tingting 29. Implement Query expansion strategy and results in custom Search Component 4/19 Done Andrej 30. Implement Pseudo Relevance Feedback in custom Search Component 4/12 - 4/25 Done Shivam 31. Calculate weights for custom ranking parameters 4/19 - 4/25 Done Shivam 32. Verify and optimize the ranking function 4/19 - 5/2 Done All
7.2 Set Up a Proxy to Access Solr Service on the Cluster In this section, we will demonstrate how to setup a proxy to access Solr service on the cluster. By following the steps below, you should be able to browse through Solr Admin UI with your local browser to reach all the indexed collections on the cluster. the following command to specify the hostname and port number for the Solr service on the cluster. This is port-forwarding via SSH, which creates a secure connection between a local computer and a remote machine through which services can be relayed. $ ssh -L 9983:solr2.dlrl:8983 <user>@hadoop.dlib.vt.edu Enter address “localhost:9983/solr” in your browser, and you should be able to see the Solr Admin UI. In the highlighted area, you can choose which collection you want to access. 7.3 Extend Solr with custom libraries Ultimately, Solr is a Java web application and as such is composed of compiled JAR containing classes to handle requests, create indexes, manage Solr cores and return search results. Additionally, Solr provides high degree of customizations as it allows its user to implement and set up custom behavior with the use of Solr plugins. This section contains information on how to configure the Solr with custom libraries and JARs. Currently, the only way to add user-written JARs to Solr is by uploading them directly to the file system of the machine running Solr service -- unlike indexes, Solr cannot import custom classes from HDFS. Additionally, if run in mode, every machine running Solr must contain a copy of the JAR files located under the same (absolute) file path. Solr loads libraries dynamically, via reflection, based on the instruction provided in solrconfig.xml. These libraries are only loaded once - when collection/core starts or restarts (is reloaded). As such, library update doesn’t require neither Solr restart nor administrative rights. Steps: Create a directory that will contain custom JAR files on all the machines running Solr service. Make sure this directory can be found at the same path for all of them.
specify the column family in the HBase table and clean_text is the specific column under clean_tweet column the Solr. _t specific the data type is _s indicates string, _f indicates float, etc. Please refer to Solr schema.xml for more details about dynamic Clean_tweet:clean_text / clean_web:text_clean_profanity clean_text_t1 Clean_tweet:collection / clean_web:collection collection_name_s1 clean_tweet:hashtags hashtags_s clean_tweet:urls / clean_web:urls urls_s1 clean_tweet:mentions mentions_s clean_web:lang lang_s clean_web:domain domain_s clean_web:title title_s doctype:doctype doctype_s tweet:archivesource archivesource_s tweet:created_at created_at_s from_user_s tweet:from_user_id from_user_id_s tweet:geo_coordinates_0 latitude_f longitude_f tweet:iso_language_code iso_lang_code_s tweet:profile_image_url profile_image_url_s tweet:source source_s time_s tweet:to_user_id to_user_id_s tweets_topics:probability_list topic_probability_list_s2 tweets_topics:topic_label topic_label_s2 cf_cf:sim_scores recommendation_sim_scores_s2 cf_cf:sim_tweets recommendation_sim_docs_s2
Sample: 2016-03-01 01:44:46.003 INFO (qtp1450821318-22) [ x:jcg] o.a.s.c.S.Request [jcg] webapp=/solr path=/select hits=4 status=0 QTime=88 In the example these are the values and their explanations. 4: Solr Log Parameters Parameter Value Date 2016-03-01 00:59:33.591 Time of query LoggingLevel INFO of logging for Log4j in Solr ServerId qtp1450821318 A unique identifier generated at the startup of Solr server and remains the same with every query. There is no information about this on Solr doc. CollectionName jcg The name of the Solr index collection/core. RequestType o.a.s.c.S.Request The type of request that has been made. This signifies the class object. Here it is class. SolrPath /solr The path of the Solr instance QueryParser select The parser that served this request. For any custom parser, this could be /custom or anything we define Query q The query being made on the Solr indexed collection