Automatic Resolution of Domain Name Disputes (2021)

Abstract: We introduce the new task of domain name dispute resolution (DNDR), that predicts the outcome of a process for resolving disputes about legal entitlement to a domain name. TheICANN UDRP establishes a mandatory arbitration process for a dispute between a trade-mark owner and a domain name registrant pertaining to a generic Top-Level Domain (gTLD) name (one ending in .COM, .ORG, .NET, etc). The nature of the problem leads to a very skewed data set, which stems from being able to register a domain name with extreme ease, very little expense, and no need to prove an entitlement to it. In this paper, we describe thetask and associated data set. We also present benchmarking results based on a range of mod-els, which show that simple baselines are in general difficult to beat due to the skewed data distribution, but in the specific case of the respondent having submitted a response, a fine-tuned BERT model offers considerable improvements over a majority-class model

## introduction domain name abuse, or cybersquatting, is the act of registering and using a domain name in bad faith. examples of such behaviour include registering a generic top-level domain (gtld) name (i.e. one ending in .com, .org, .net, etc) that incorporates another's trademark, with the intent to on-sell it to the trademark owner; to dupe the public into erroneously assuming that the owner of the domain name is the trademark owner; or to interfere with the trademark owner's business, to name a few. a victim of cybersquatting has a means to remedy the situation. in 1999, the agency responsible for the internet's naming system, the internet corporation for assigned names and numbers (icann), introduced a fast and low-cost mandatory arbitration system as an alternative to the slow and expensive litigation of cybersquatting disputes -the uniform domain name dispute resolution policy (udrp). 1 since its establishment, approximately 80,000 complaints have been resolved through the process @xcite . udrp cases, over 90% of which are in english, are decided by an arbitration panel comprising either one or three experienced trademark lawyers. the typical arbitrator's fee of usd 1,000 is small relative to the time expended and the usual hourly rate for legal services of this type. the performance of the task is, in effect, cross-subsidized by the arbitrator's day job. we introduce this task as a means to assist the service providers and arbitrators by automatically pre-assessing complaints. the ultimate aim is to identify those complaints whose characteristics are predictive of success. doing so has the potential to aid in the optimal allocation of finite resources -in this case, the time and expertise of the arbitrator -because such cases should be relatively easier for the arbitrator to decide. by identifying the easier cases, service providers and arbitrators can direct more resources towards the remainder of complaints, with the result that the cases most in need of resources are given a greater share of time. that should improve the quality of decision-making, benefiting all stakeholders in the domain name system. the distribution of outcomes under the udrp is highly skewed. of the complaints that proceed to a panel determination, 87% succeed. however, the distribution of cases is also highly skewed by whether or not a response to the complaint is filed. in the arbitration process, the domain name registrant has the right to file a rebuttal (a "response") to the complainant's claim. for nearly three-quarters of all complaints, no response is filed, in which case the complaint almost always (94% of the time) succeeds. where a response is filed, however, the success rate is only 66% @xcite . it follows that cases where a response is filed are of particular interest to the task. while we are by no means the first to propose the task of legal judgment prediction @xcite @xcite , previous work has focused almost exclusively on criminal or human rights cases, which we argue have sensitivities and ethical dimensions that are yet to be fully quantified @xcite . part of our intention with this paper is to introduce a "lower-stakes" task/data set in terms of its direct and indirect implications for individuals and civil liberties, which is far from "solved" in terms of current nlp capabilities. in the introduction of this new task, we outline in section 2 the process of how a complaint is dealt with once it is submitted to an icann-accredited service provider, and how a decision on the complaint is reached by the arbitrator appointed to decide it. in this study, we ultimately aim to ascertain if providing predicted outcomes to the arbitrators will assist them in the dispute resolution process -not as a means to make the decision for them, but as a way to pre-sort the difficult cases from the straightforward ones. in section 3, we describe how we develop our corpus for the dndr data set, and section 4 details how we prepare this data for our experiments as well as the set-up of the fine-tuning experiments. given that the data set is highly skewed towards one class, we expect that the reporting of the overall binary task to not impressively exceed a majority-class baseline. however, the results from the important partitioning of the data between cases with and without responses in section 5 shows that this task has the potential to aid arbitrators in performing their duties. in section 6, we discuss how we foresee ai to be of use in legal processes and assisting legal professionals in their decision-making, and why it's important to define best practice when it comes to employing artificial intelligence to assist in the legal domain. ## background a domain name dispute commences when a trademark owner ("complainant") files a complaint with an icann-accredited service provider, seeking a remedy against a domain name registrant ("respondent"). of the six bodies accredited by icann as udrp service providers, foot_1 the largest by far is the world intellectual property organisation (wipo). over 52,000 cases -more than one half all udrp disputes -have been filed with wipo @xcite . its case-load is growing, with filings in 2020 topping 4,000. to obtain a remedy, the complainant must show that four conditions are satisfied: (1) the complainant owns a trademark; (2) the domain name is identical or confusingly similar to that trademark; upon the filing of a complaint, wipo checks it for compliance with administrative requirements, obtains certain information about the domain name and the registrant of it, then notifies the complaint to the respondent. the respondent has 20 business days in which to file a response. once the deadline for filing a response has passed, wipo appoints a panel and supplies it with the case file -essentially, the complaint and any response. the panel has 14 days in which to decide the case and to write a decision justifying its outcome, which are both publicly disclosed @xcite . a panel decision will generally come to one of two outcomes: either the complaint succeeds, or the complaint fails (is "denied"). in a small number of cases, where the parties settle the dispute after commencement but prior to a decision being rendered, the panel will "terminate" the complaint. where the complaint succeeds, the panel will order one of two remedies, based on the desires of the complainant: either cancellation of the domain name, or transfer of the domain name to the complainant. the registrar of the domain name will give effect to the order (i.e. either cancel or transfer the domain name) unless the respondent challenges the decision by filing a court case in the appropriate jurisdiction @xcite , something which almost never happens. thus, in practice, the panel's decision is final and binding. ideally the task would be undertaken on the complaint and any response that was filed. unfortunately, however, neither of those documents are published. instead we use the panelist's decision, which is publicly providers-6d-2012-02-25-en 3 for full statistics, see https://www.wipo.int/ amc/en/domains/statistics/outcome.jsp available. wipo decisions are highly standardized in structure and form. they typically contain seven sections, with the following headings: (1) "the parties"; (2) "the domain name and registrar"; (3) "procedural history"; (4) "factual background"; (5) "parties' contentions"; (6) "discussion and findings"; and (7) "decision". sections ( 4 ) and ( 5 ) are the panel's summaries of the facts, and the arguments about the legal import of those facts, respectively, that are contained in the complaint and response. they are, therefore, a proxy for the information that is contained in the complaint and response, which are not publicly available. ## corpus creation the corpus was derived by the scraping the text from the published decision of each gtld domain name case foot_2 for the years 2000 to mid-2020. fig- for this preliminary work, we decided to only keep the outcome labels that had enough instances to be able to reasonably train a model, which is why we only kept the top 3 labels: transfer, complaint denied, and cancellation (which account for over 99% of the data), as shown in the purpose of the task is to predict the outcome of the case based on the complainant's grievance and the respondent's counter to it. as detailed in section 2, the grievance in the form in which it was filed with the service provider (the complaint) is not published. this is also the case for the respondent's counter to the grievance (the response). however, the contents of both the complaint and the response are summarized by the panel in the sections on "factual background" and "parties' contentions". we retain those sections, as a proxy for the facts alleged by complainant and the respondent, and for their legal arguments pertaining to those facts. we also retain the three preceding sections of the decision. the first two of these are "the parties" and "the domain name and registrar", in { "case-0": { "status": "transfer", "url": " https://www.wipo.int/amc/en/domains/search/text.jsp?@xmath0, "no": "d2020-0001", "complainant": "groupon, inc.", "respondent": "domain admin, whois privacy corp.", "title": "", "text": { "section-3": { "h3": "3. procedural history", "p": [ "the complaint was filed with the wipo arbitration and mediation center... ], "a": [] }, "section-4": { "h3": "4. factual background", "p": [ "the complainant is a major player in local commerce that offers consumers a... ], "a": [] }, "section-5": { "h3": "5. parties' contentions", "p": [], "a": [] }, "section-6": { "h4": "a. complainant", "p": [ "the complainant contends that the disputed domain name is confusingly similar to... ], "a": [] }, "section-7": { "h4": "b. respondent", "p": which the complainant, the respondent, the domain name, and the registrar are named. the third is the "procedural history", in which is recorded the dates on which the complaint and response were filed, the date on which the panel was appointed, and the name of the panelist(s). the final two sections of the decision are "discussion and findings", in which the panel analyzes whether or not the complainant has proven its claim; and "decision", in which the outcome is formally recorded. we remove those sections, as they contain the outcome of the case, which is the very thing the task is seeking to predict. in total there were 33,841 cases at the time of scraping. however, not all of them could be collected. there are four cases that are not available, and 67 cases cannot be scraped due to inconsistencies in the html tags. on top of that, there are 2,757 non-english cases. so, after the filtering, 31,013 cases remained. we filtered the data further by selecting cases that have both procedural history and discussion and findings sections. moreover, we only chose cases that have transfer, complaint denied, and cancellation decision. in the end we obtained 30,311 cases. this data set only contains procedural history, factual background, and parties' contentions sections from each cases. the vocabulary size of the data set is roughly 245k unique tokens. the data set is split into training, development, and testing set. their length on average is 1,131, 940, and 939 words respectively. however, these numbers are reduced after performing text processing before training the model. we provide a data statement @xcite for the data set in the appendix. ## experiments we conducted a series of experiments on our new data set. we developed 2 rnn models, 3 bert models, as well as 2 non-neural models. given that the data set is highly imbalanced, we also provide a majority-class baseline. ## related work and discussion this is the first work to propose predicting judgments of ip cases, in the low-stakes (but technically challenging, as evidenced by the empirical results) setting of world intellectual property organisation domain name resolution cases. while the legal context for this research and resulting data set are novel, we are in no way the first to propose judgment prediction as a task for legal nlp @xcite @xcite @xcite . perhaps the most popular judgment-related data set is the chinese ai and law challenge data set ("cail 2018"), where the task is to predict which of 202 charges the accused is guilty of as a multi-label classification task, based on more than 2.6m criminal cases published by the supreme people's court of china in mandarin chinese @xcite . the original competition attracted over 200 submitting teams @xcite , and included two other tasks: (1) law articles: identify criminal law articles that are cited as being relevant to a given case; and (2) sentencing: predict the jail sentence, in months, for the guilty party. the data set has since been used widely as a benchmark data set for reasoning and legal nlp @xcite @xcite . for english, the most popular data set is perhaps that of @xcite based on over 11k european court of human rights ("ecthr") cases, in the form of two tasks: (1) a binary classification task: does the case violate any of the 66 articles and protocols of the ecthr; and (2) a multi-label classification task: which, if any, of the 66 articles and protocols does a given case violate. 8 other data sets include a french data set of 126k cases from the french supreme court @xcite , an english data set of 5k cases from the uk court of appeal @xcite , an english data set of 28k cases from the supreme court of the united states @xcite , and a german data set of around 6k tax law appeal cases from the german fiscal courts @xcite . while the majority of this work has focused solely on the task of predicting the case judgment, @xcite equally focus on using the trained models to analyze the fairness of judicial decisions, and more generally as an analytic tool for enhancing understanding of how judicial decisions are made. methodologically speaking, much of the past work has approached judgment prediction using off-the-shelf topic classification methods (e.g. using tf-idf-weighted bag-of-words document representations with a linear-kernel svm or naive bayes model, similar to our logreg and svm), although recently the data sets of @xcite and @xcite have equally been used as benchmark data sets to evaluate longer-context pre-trained lms over (based on the fact that the average document length tends to be much longer than the standard 512 token limitation of conventional bert models). as alluded to in the introduction, @xcite recently questioned the appropriateness of data sets such as @xcite , on the grounds of issues such as there being only superficial anonymisation of the data (leading to risk of the model learning demographic biases based on names and place mentions), privacy concerns (e.g. lms fine-tuned on the data would potentially generate outputs containing sensitive information regarding defendants mentioned in the data), the data set being developed without engagement with the relevant legal authorities, and the appropriateness of the criminal domain for the deployment of automatic decision-making tools, given the high personal stakes for defendants. while noting some of the counter-examples to these concerns of tsarapatsanis and @xcite and the need for further debate regarding the ethics of the applications of legal nlp, we return to our earlier claim that legal areas such as domain name disputes are a more appropriate short-term application domain for judgment prediction, in the sense that there are very few personal sensitivities which would have implications on the later life of the individuals represented in the data. wipo documents are relatively consistent in formatting and content between cases, similarly to the ecthr documents, e.g., which follow a rigid format. the wipo documents are generally more consistent than other legal cases, in which there is a significant issue for nlp techniques, as discussed by @xcite . ## conclusion we have introduced a new large-scale data set for legal judgment prediction, derived from domain name dispute cases from the world intellectual property organisation where a complainant has applied for a domain name to be transferred or cancelled on the grounds of trademark infringement. the data set is released with a data statement as defined by @xcite in the appendix. in the context of binary classification of the complaint being accepted or denied, we provided results for a number of benchmark nlp methods. in terms of macro-averaged f-score (to counter the effects of class imbalance), we showed that bert with fine-tuning performed the best, including over the subset of cases where a response has been submitted. however, we equally showed that there is considerable room for improvement in the results, and plenty of room for further work on this task. in terms of future work, we aim to further build on the models presented in this paper, and test their utility as decision support models. our initial design was to simply experiment with the threshold of the model's output as a means to trade off precision for recall, and rely on the probability score as a proxy for model confidence and a measure of reliability. however, pretrained models such as bert and legalbert can suffer from severe miscalibration due to over-parameterisation. @xcite . as a first step, we aim to perform model calibration experiments to address the over-confidence issues commonly seen in these fine-tuned models. further to this step, we will validate the accuracy of these calibrated models by revisiting the errors made by the system and verifying the original judgments by the panelists with a fresh set of legal eyes.