Who wins in consumer arbitration? Historically, this question has been nearly impossible to answer, as most arbitration proceedings are a private black box, and arbitral forums release only limited summary statistics. One exception is the Financial Industry Regulatory Authority (FINRA), which arbitrates virtually all disputes between investors and stockbroker-dealers, and makes all of its nearly 60,000 written arbitration decisions publicly available in an online database. This Article is the first to use computational text analysis tools to study these decisions, and to construct a measure of the claimants’ win, loss, and settlement rates. It is the first installment in an original data analytics project that aggregates dispersed public data and document sets to assess the efficacy of arbitration outcome transparency as an investment protection measure. This Article makes three main contributions. First, the results of our novel study provide a more granular picture of customer experiences in the FINRA forum. We identify settlement as the most frequent outcome, followed by claimant losses, and then wins. In twenty percent of cases, we identify the presence of multiple outcomes per arbitration decision, where a claimant lost some claims but won or settled others, for example. This suggests a greater complexity and nuance in the notion of investor success than FINRA’s monetary recovery versus no monetary recovery outcome measure—and previous scholarship—have recognized. Second, we discovered that the structure of FINRA’s written arbitration decisions prevents further exploration of the amounts of compensatory damages that claimants recover, if any, compared to the amounts requested. Our final contribution is, therefore, a set of recommendations to FINRA—applicable to other private dispute resolution forums as well—to increase data access, usability, and transparency.
Introduction
Neil Harrison worked as a stockbroker-dealer in the early 2000s at a string of investment firms and brokerage houses.1 He invested money and traded securities on his clients’ behalf and advised them about their financial future.2 He also diverted his clients’ money to “support his drinking and gambling habits;”3 sold stock without investors’ permission;4 induced clients, some of whom were over eighty years old, to lend him money;5 and committed various other violations of securities law. One client, a sixty-seven-year-old, self-employed truck driver, lost his entire retirement savings due to Harrison’s fraud.6
Eventually, Harrison was caught, indicted for securities and mail fraud, and served twenty-one months in prison.7 Illinois securities regulators also fined him for taking illicit loans and improperly selling unregistered securities.8 In 2010, Harrison was finally expelled from the securities industry.9
What about his clients? Like virtually all stockbroker clients, they would have been barred from taking Harrison to court under the terms of pre-dispute arbitration agreements (PDAAs), signed as a condition of investing with Harrison and his investment firm employers.10 Under PDAAs, the overwhelming majority of wronged investors’ claims must be resolved via binding private arbitration administered by the Financial Industry Regulatory Authority, or FINRA.11
Two Harrison clients did, in fact, pursue FINRA arbitration. The first claimed about $66,000 in losses due to Harrison’s bad acts.12 After arbitration, she won $45,000 in compensatory damages, plus interest and costs.13 The second investor made claims against Harrison, his direct supervisor, and his investment firm employer.14 Originally seeking $400,000 in damages, the investor ultimately won only two dollars, one dollar in compensatory damages from Harrison and one from the firm, plus attorneys’ fees and costs.15
In statistics published by FINRA, both outcomes appear within the same outcome measure—monetary recovery, which lay persons often equate with a successful outcome or “win”—as both investors recovered some amount of compensatory damages above zero.16 Yet, while the first investor recovered 68% of the compensatory damages she requested, the second recovered only 0.0005% of his original claimed losses.17
Whether these recoveries should count as “wins”—whether they truly compensated the claimants for their losses at Harrison’s hands and, more generally, whether the process met its investor protection goals—are complex questions that require more than a simple tally of non-zero recoveries, the metric FINRA uses to record results in its arbitral forum.18 Indeed, scholars have rightly pointed out that work to classify the outcome of an arbitration as a binary win or loss based solely on whether any monetary sum has been recovered tells us little about what happened or whether the forum is fair to consumers.19
This Article employs a set of computational text analysis tools to study over 3,000 publicly available FINRA arbitration decisions, issued between 2013 and 2018, in disputes between investors and broker-dealers to build out a more nuanced and granular taxonomy of FINRA arbitration outcomes.20 As an example, we find that about one-fifth of the awards21 in our data set contained multiple outcomes (win/loss/settlement), suggesting that the presence of a positive dollar amount may both over- and under-state consumer22 success, and an analysis of any one award is not complete without also looking for characteristics of multiple outcome types.
This Article is the first installment in an original and ambitious data analytics project that aggregates dispersed public data and document sets to assess the efficacy of FINRA’s investor protection measures for investors like the Harrison clients. This work takes advantage of FINRA’s unusual transparency as an arbitration forum. Arbitration has a reputation as a black box, from which scant information escapes about parties’ experiences and proceedings’ outcomes.23 FINRA, however, makes all written arbitration decisions publicly available on its website, along with limited summary statistics.24 It is, therefore, no surprise that scholars look to FINRA’s awards database, the resource from which the Harrison clients’ respective awards were drawn, in efforts to assess consumer outcomes in the forum and the factors that might lead to them.25 Our work builds on this previous scholarship, but deploys a new set of tools and a unique focus. Using modern data analytics techniques, we can pry open the arbitral black box, at least part way, and develop a more textured understanding of customer investor experiences in FINRA proceedings.26
In this Article, we also document the computational difficulties we encountered in extracting information from the text of arbitration decisions. On this basis, we offer a set of data access recommendations, for regulators and decisionmakers within and outside the FINRA forum, for improving awards’ structure to make them more conducive to measurement and analysis. These findings have application beyond FINRA to all types of dispute resolution and can help shape data collection and reporting standards to better facilitate independent research and public understanding.
The Article proceeds in five Parts. Part I begins with an overview of arbitration, including common support and criticism of the now-ubiquitous dispute resolution mechanism. FINRA’s arbitral forum is next presented as a case study due to its unique features. We start by describing the history of arbitration in the resolution of securities disputes, moving from an initially voluntary mechanism to a functionally mandatory process for the resolution of all disputes involving stockbrokers. We next describe FINRA’s broad jurisdiction over such claims before focusing on one claim type: investor claims against their stockbrokers and the varying procedures and options available to investor-claimants, dependent on claim size. Part I then moves to a detailed discussion of one characteristic that sets FINRA dispute resolution apart from other arbitration forums: the plethora of public information, dispersed across sources, concerning consumer experiences in FINRA arbitration.
In Part II, we introduce the proposition that text analytics tools can provide multiple frames through which a final arbitration decision can be defined as a win, loss, settlement, or some combination of the three. We then describe the methodology for assembling our approximately 3,000-award set and analyzing each decision’s text to classify its outcome. Specifically, we wrote code to extract all dollar amounts from the text, assembled keyword counts, identified the terms that best distinguished one outcome from another, and used a technique called sentiment analysis that scores text by its positivity or negativity. Throughout, we validated our results by hand-checking samples of results for accuracy and compared our own outcome tallies to those published by FINRA. We also experimented—unsuccessfully, for now—with building a machine learning classifier to algorithmically predict a dispute’s outcome based on decision text, and with clustering methods to uncover latent groupings within the set of arbitration decisions.27 Work on those projects remains ongoing.
We then turn, in Part III, to the results of our analysis. By using targeted term frequencies, we were able to classify eighty percent of awards as having a single outcome of win, loss, or settlement. The remaining twenty percent reflected multiple outcomes within a single award, including a win, loss, and/or settlement. This exercise elucidated greater nuance of consumer experiences than reflected in FINRA’s binary classification of awards by virtue of whether any damages were recovered or not. Sentiment analysis, however, was not capable of measuring an award as a win, loss, settlement, or variety thereof. Finally, using targeted dollar amount extraction to measure the ratio of the amount an investor was awarded to the amount they sought was not feasible due to complexities within the data.
Thus, in Part IV, we offer a set of data access recommendations for increasing the utility and transparency of FINRA’s awards database and summary statistics. We end by exploring the conclusions that can be drawn from our analysis. We also mark a path forward, describing our future research that will continue to probe the extent to which the FINRA arbitration forum is serving its stated investor protection and transparency aims.
I. Arbitration and Securities Claims
A. Arbitration: Ubiquitous and Criticized
Though it took congressional action to start,28 the arbitration revolution has been fully embraced by courts formerly reticent to enforce PDAAs,29 with recent Supreme Court decisions supporting scholars’ assessments that arbitration will continue to reign supreme in the resolution of future consumer disputes.30 Arbitration clauses prevail in consumer contracts,31 from credit cards,32 to auto loans,33 to mobile providers.34 Employment agreements regularly contain arbitration clauses such that an employee may not publicly resolve a dispute with an employer in court.35 Though arbitration clauses predominate in consumer contracts, the same companies that require consumers to arbitrate quite ironically themselves prefer litigation to resolve disputes with their peers.36
Arbitration purports to fill a necessary gap, providing consumers with a more economical and efficient process for resolving their claims, particularly when the claim amount is relatively small.37 Despite these purported benefits, mandatory arbitration has received much criticism.38 Criticism of arbitration arises both from its impact on dispute resolution writ large, as well as its effect on individual claimants.
At the systemic level, critics bemoan the loss of the civil jury trial caused by the concurrent uptick in claims that must be arbitrated.39 This critique is particularly acute, as researchers have found that most consumers are unaware of arbitration clauses or, if they are aware, how they operate.40 If a consumer does understand that they are bound by a PDAA, they rarely understand the limitations such a clause places on their ability to pursue claims that might arise after agreeing to the arbitration provision.41 Moreover, as arbitrators’ decisions do not have precedential value, those industries in which arbitration predominates have been said to stagnate, as there is no movement or change within the law.42 Finally, certain rights, including those that Congress intended to be enforced by the public, may go unenforced due to arbitration’s intervention in the area.43
Arbitration also has individual impacts, the aggregation of which may raise larger systemic concerns. For example, scholars argue that the rise in the ubiquity of arbitration, coupled with its lack of transparency, has led to consumers not pursuing valid claims at all.44 Moreover, scholarship suggests that while PDAAs now regularly bar class actions, complex claims are not well suited for individual arbitration, and those claims may, likewise, not be pursued.45 Others argue that the arbitration costs are too high and operate to deny consumers the ability to obtain redress.46 Moreover, the confidentiality and lack of transparency into most arbitral forums make it difficult for patterns to emerge, thus discouraging individual litigants from enforcing their own rights or obscuring industry-wide concerns that regulators or law enforcement might wish to pursue.47 On the procedural front, arbitration is also criticized, particularly for the repeat player effect, which posits that members of an industry in which PDAAs predominate have a better chance of success in the forum than individual consumers who have substantially less frequent engagement with the forum.48 Recent scholarship bears out this impact, indicating, however, that the repeat player phenomenon can benefit individual consumers to the extent they employ an attorney with significant experience within the forum.49
Beneath these critiques of arbitration runs an undercurrent suggesting that the confidentiality at the core of arbitration creates such a high wall shielding the industries it covers that arbitration is fair neither to the public at large nor to unsophisticated individuals.50 Empirical studies of arbitration are limited in their reach as a result of the lack of information on what is happening within the forum, producing many studies simply of whether or not consumers prevail, and others raising questions as to whether measuring winning and losing actually tells us anything about consumers’ experiences.51
It is against this backdrop that arbitration within one industry arises as distinct from most of its peers. FINRA now hosts a single arbitral forum that resolves more securities arbitration claims than any other forum.52 FINRA’s arbitration process, born out of necessity to facilitate trust in the securities markets,53 makes public an uncharacteristically high level of information involving the claims made within the forum or otherwise levied against a stockbroker, potentially providing a level of insight into consumer arbitration not typically available.54
B. Arbitration of Securities Disputes: From Voluntary to Mandatory
Arbitration has existed in the securities realm nearly as long as securities have been traded in this country, with its roots being traced back to the late 1700s.55 Examining the history of securities arbitration in the United States, Jill Gross argues that securities exchanges accepted consumer-investor complaints against stockbrokers in their arbitral forums as a consumer protection mechanism deriving from the exchanges’ need to ensure trust and confidence in their industry.56 Without a means to quickly and fairly resolve disputes with stockbrokers in rapidly moving financial markets, investors would not trust the markets or the individuals serving them within them.57 While acknowledging the consumer-friendly and consumer-trust building aspects of securities arbitration, others suggest that the securities exchanges also provided for arbitration as a mechanism for their members to enforce those contracts that were not otherwise enforceable at law.58 Thus, certain contracts disfavored by courts or at law could be enforced by a purchaser against a stockbroker in an exchange-sponsored arbitration.59
Despite arbitration’s role as a mechanism to induce investors to trust the securities markets and enforce investment contracts,60 PDAAs purporting to require consumer-investors to arbitrate claims against their stockbrokers were held unenforceable by the Supreme Court for decades after the enactment of the 1925 Federal Arbitration Act.61 It was not until the mid-to-late 1980s that a series of decisions changed course and ultimately found PDAAs in the securities industry enforceable.62 As a result, the previously voluntary securities arbitration processes housed by the various stock exchanges, or self-regulatory organizations (SROs), would become essentially mandatory, with each of the SROs witnessing a significant increase in the number of claims heard and decided in their respective arbitral forums soon thereafter.63 Since then, through industry consolidation, FINRA has become the sole SRO-sponsored arbitral forum.64
The claims heard within the FINRA forum encompass nearly all interactions between broker-dealers65 and the investors who hired them,66 known in FINRA parlance as customers.67 FINRA also administers intra-industry disputes, i.e., between industry members68 and their associated persons—employment disputes between stockbrokers and investment firms, for example.69 Industry disputes and customer disputes are resolved under separate, yet substantially similar, codes of arbitration procedure.70 FINRA also offers a mediation program for disputes if the parties voluntarily submit to it.71
Most customer disputes arbitrated by FINRA come to the forum as a result of a PDAA contained in a brokerage agreement between the consumer-investor and the broker-dealer firm, leading many to describe FINRA customer arbitration as mandatory.72 Even if a customer’s dispute with a broker-dealer or associated person is not required to be submitted to FINRA arbitration as a result of a PDAA, the claim can nevertheless be arbitrated.73 A customer has a unilateral right to request arbitration on their own accord, and the member/associated person is required to arbitrate by virtue of their association with FINRA.74 Alternatively, if the parties enter into a post-dispute arbitration agreement, FINRA will administer the dispute.75
Parties may submit an extremely wide range of conduct and disputes to FINRA arbitration. The customer code permits parties to submit claims that “arise[] in connection with the business activities of the member or the associated person, except disputes involving the insurance business activities of a member that is also an insurance company.”76 There are, however, two major exceptions to the broad range of cases eligible for arbitration before FINRA.77 First, FINRA will not accept class or collective action claims in arbitration, whether raised in the industry or customer context.78 Second, shareholder derivative claims are excluded from FINRA customer and industry arbitration.79 FINRA’s Customer and Industry Codes largely mirror each other, and for the remainder of this piece, we limit our analysis to customer arbitration proceedings and the relevant provisions of the Customer Code of Arbitration Procedure.
C. Customer Arbitration Proceedings in the FINRA Forum
Depending upon the size of their claim, customer-claimants80 can elect from among three different FINRA arbitration paths.81 Claimants with the smallest claims, $50,000 or less, have the greatest range of options under the so-called simplified arbitration provisions.82 By default, claimants seeking $50,000 in damages or less will have what is known as a paper proceeding, a process by which the claim is determined entirely on the pleadings and written evidentiary briefs submitted by the parties.83 Paper proceedings under the simplified arbitration rules were developed to be simpler and less costly for investor-claimants to navigate84 and typically lead to a final decision much more quickly than a traditional FINRA proceeding decided after a full hearing.85 Theoretically, at least, an investor could navigate such a proceeding on their own without legal counsel, an unfortunate reality given the difficulty claimants have in securing counsel for smaller claims.86 Paper proceedings are decided by one arbitrator, who is selected from the public87 chairperson88 roster.89 Though parties in a paper proceeding have the ability to obtain some discovery,90 paper proceedings are exempted from the mandatory discovery provisions requiring parties to produce certain documents in full hearing cases.91 Paper proceedings represent a very small portion of FINRA proceedings overall, from two to five percent of decided cases between 2015 and 2019,92 and FINRA reports that customers obtained some recovery—more than $0—in such cases at a rate of between 31% and 48% in recent years.93
While paper proceedings might be preferable for some customers, claimants with smaller claims have great flexibility to elect into a hearing if they wish.94 Notably, in customer arbitration, election out of the paper proceeding default is an option only the customer, and not an industry member, can invoke.95 Customers can choose two different hearing options: (1) a special telephonic proceeding; or (2) a traditional hearing subject to all provisions of the Customer Code.96 The special telephonic proceeding option is a new addition to the FINRA forum and is a hybrid between a paper proceeding and a full, traditional hearing.97 In this proceeding type, a telephonic hearing is held and each side has the ability to provide testimony in an abbreviated single hearing session, although cross-examination is not permitted.98
The special proceeding option was intended to provide customer claimants with the opportunity to tell their story and engage with an arbitrator while retaining the consumer-friendly aspects of paper proceedings, such as lower cost, ease of use, and efficiency.99 Customers in a special proceeding are not subject to cross-examination by an opponent, though, with that potential benefit, they are similarly limited in their ability to call and cross-examine the respondent or witnesses.100 Special proceeding elections have thus far been rare, with only eight such cases decided from September 2018 until the end of 2019, and customer-claimants have not recovered any damages in all but one of those cases.101 The time from filing through resolution of a special proceeding is slightly longer than a paper case, at an average of seven months.102
The vast majority of claims filed proceed under the regular hearing rules, either by the election of a claimant with a smaller claim or because the damages sought exceed $50,000 and the parties have no option other than a traditional arbitration hearing.103 Customers with claims exceeding $50,000 proceed to a full arbitration with prehearing,104 discovery,105 and hearing provisions.106 Claims seeking up to $100,000 are limited in one regard: they are heard and decided by a single, chair-qualified arbitrator.107 Claims exceeding $100,000 are heard and decided by a panel of three arbitrators,108 with customers able to elect an all-public panel.109 A full hearing is conducted in person and without limits as to its ultimate length or upon calling witnesses and cross-examining an opponent’s witnesses.110 A typical customer hearing case takes approximately fourteen months from filing to final decision, with customers receiving some monetary recovery in thirty-eight to forty-five percent of such cases from 2014 through the end of 2019.111
D. Sources of Information Concerning FINRA Arbitration
As the above discussion of how frequently each type of FINRA arbitration process is invoked, its average resolution time, and the percentage of proceedings in which a customer recovers damages112 indicates, much information concerning FINRA customer claims is available to the public.113 While unique in the number of categories of information provided, FINRA is not unique in providing consumer monetary recovery rates and some basic information concerning proceedings in its arbitral forum.114 For example, California and Maryland require arbitral forums to provide information concerning mandatory arbitration.115 Thus, the American Arbitration Association (AAA) and JAMS provide spreadsheets detailing basic information about consumer and employment cases decided within their respective forums, including the name of the non-consumer respondent, the name of the consumer’s attorney or whether they are self-represented, filing and disposition dates, how the case was closed, the name of the neutral, and the amount of the award.116 FINRA’s dispute resolution statistics provide similar aggregate information.117
What sets FINRA apart from arbitral forums like AAA or JAMS is that, in addition to publicizing summary statistics and basic information about decisions rendered,118 FINRA makes available, through multiple media, raw information concerning individual customer complaints against broker-dealers and their associated persons, as well as the resolution of such claims.119 It is here that FINRA’s multi-dimensional responsibilities as a regulator and licensor of broker-dealer firms and their associated persons, overseer of the markets, and host of a dispute resolution forum could potentially provide a full picture of investor-customer experiences when a dispute with a stockbroker arises.120
Most intuitively related to the study of consumer experiences in arbitration is FINRA’s rare choice to make the final written decision in every arbitration proceeding that concludes after a hearing publicly available121 in a searchable database available on its website.122 Included in the awards database are customer claims, industry claims, and expungement actions, proceedings in which a broker seeks to remove a customer’s complaint from the broker’s regulatory record.123 These case disposition documents, counterintuitively known as “awards” whether or not the claimant recovers anything,124 follow a largely uniform pattern and include certain required information.125 For example, among other items, every FINRA award must list the parties’ names, their counsel or other representatives, a summary of the matters presented in the claim, damages requested and awarded, the arbitrators’ names, the dates the claim was filed and closed via an award, and the location of the hearing.126 An award may, but is not required to, contain a rationale describing how the panel reached its ultimate decision in the case.127 If, however, the parties jointly agree, the panel is required to issue an explained decision “stating the general reason(s) for the arbitrators’ decision”128 in both of the hearing option cases.129 Explained awards are extremely rare in the FINRA forum.130
Given the depth of information contained in the FINRA awards database, it is no surprise that scholars have relied upon it on numerous occasions to answer various questions concerning customer experiences in arbitration,131 the expungement of consumer complaints from brokers’ records,132 and employment disputes.133 No previous research, however, has used computational text analysis to study a comprehensive set of FINRA awards to assess different outcome measures—the subject of the present Article and the jumping-off point for the authors’ future stream of related research.
II. Hypothesis, Methodology, and Data
We begin our work with the hypothesis that using modern text analytics tools to assess results in FINRA awards would produce a useful taxonomy of outcomes.134 We further hypothesize that the results of such an exercise would provide a more textured understanding of why reliance on any one measure of success might be unreliable or tell an incomplete story as to the outcome. Relatedly, we hope to encourage researchers—and FINRA itself—to use more than one conception of success as a frame through which to view arbitration outcomes.
To test our hypotheses, we first assembled a set, or corpus, of documents to study, determined the analytics techniques best suited to our aims, and constructed a methodology that would allow us to deploy such techniques to extract meaning from the corpus text. We describe each of those steps in the Sections that follow.
A. Assembling the Corpus
To assemble our corpus, we wrote code to automate the process of downloading all arbitration awards available from FINRA’s publicly available online awards database.135 At the time of this mass download, 55,655 files were available in .pdf format, spanning March 1, 1998 through May 10, 2018—the date on which we ran the download.136 Along with the award files themselves, we scraped all information, or metadata, returned by the database queries that described each award.137 Using the metadata, we eliminated any documents that were not related to a FINRA proceeding.138 We further removed any document that was not coded as an “award,” thereby eliminating other document types that are related to FINRA’s dispute resolution processes but do not reflect a final arbitration decision. We then limited our data set to a complete five-year period—May 1, 2013 through May 1, 2018—using the “DOA” (Date of Award) field.139 Our resulting data set was a corpus of 6,354 FINRA arbitration awards.
Because this corpus included both industry cases (broker versus investment firm) and customer cases (investor versus broker), we took further steps to identify and isolate the customer cases. This filtering is consistent with our overarching research aim: to aggregate dispersed public data and document sets to assess the efficacy of FINRA’s investor protection measures. After converting all files to machine-readable text format, we wrote code to extract the text labeled “Nature of Claim” within each award. After extensive manual review of the text output, we identified 3,227 customer awards issued in disputes between investors and stockbrokers. This set of 3,227 became the study set upon which we conducted the remainder of our analysis.140
B. Text Analytics Tools
In general terms, text analytics refers to the use of computational tools to extract meaning from unstructured text.141 Also known as natural language processing,142 text analytics enables researchers to automate the process of turning text into data—in our case, transforming over 3,000 written arbitration decisions, composed of over 5.3 million words—into an organized data set susceptible of analysis, without having to read each and every decision and extract the relevant information by hand.
In this project, we used an array of text analytics tools: segmentation, word counts, sentence parsing, targeted term frequencies, keyness measures, sentiment analysis, and targeted dollar amount extraction. We also experimented with creating a machine learning classifier for outcome prediction purposes and with clustering to identify natural groupings within the corpus; this work is ongoing.143 Throughout, we used text analytics and statistical packages implemented in the programming language R.144 Specific packages are identified in the footnotes; scripts and data sets are available from the authors upon request.
Each method is described thoroughly, along with our results, in Part III below. First, however, the following Section introduces our corpus of 3,227 investor versus stockbroker arbitration awards in more detail by providing some simple descriptive statistics.
C. Descriptive Statistics
Figure 1 below illustrates the distribution of the awards over our five-year study period, May 1, 2013 through May 1, 2018.145
As Figure 1 shows, the earlier awards outnumber the later ones, and 2018 is underrepresented due to the May 1 cutoff.146 However, our analysis pools the award data from all years, and does not attempt to uncover longitudinal or time series trends. In fact, with some exceptions described further below, the textual characteristics and outcome distributions within the award set were remarkably consistent year over year in our study period, justifying our pooling of the five years. Future research may expand our timeframe to investigate the possibility of time trends over a longer period of years.
The awards in our corpus were, on the whole, relatively short. The average award length was about 1,600 words, or the equivalent of just over three single-spaced pages. Recalling the discussion above of explained versus unexplained awards, this short average award length suggests that longer, explained awards were rare in our corpus, as is true for the FINRA forum in general.147
Interestingly, as Figure 2 below shows, average award length in our corpus increased over time, from about 1,650 in the 2013 awards to almost 2,050 in the 2018 awards. Further work will investigate this trend, including exploring whether explained awards were overrepresented in the later years, whether FINRA’s published guidance on expungement in 2017 resulted in longer awards, or whether the forum’s unexplained, non-expungement awards merely increased in length.148
This length trend does not affect our analysis of award outcomes, however, as our analytic techniques described below do not depend on whole-document word counts or award length. Moreover, as Figure 3 below illustrates, the length of each section of the awards, as a proportion of total word counts, was roughly the same year over year. Much of our analysis below was conducted on the “Relief Requested,” “Award,” and “Other Issues Considered and Decided” sections of the arbitration decisions, isolated from the remainder of the text.149 Figure 3 confirms that those three sections remained a relatively consistent length, as both an absolute and relative matter, throughout the study period.
We now turn to our analyses, using an array of text analytics techniques to produce a more granular, useful taxonomy of FINRA arbitration outcomes in disputes between investors and stockbrokers.
III. Defining a Win? Applying Text Analytics Techniques to Determine Outcome(s) in FINRA Awards
If our hypotheses are correct, a computational text analysis approach would result in a taxonomy of outcomes in FINRA arbitration proceedings that proceed to a hearing. While our results would not explain the factors that were most predictive of success,150 the differing techniques and conceptions of a win would help provide a more textured understanding of success and shed light upon the limitations that can result when relying on one framework versus another.
Our experience in applying these techniques to the awards corpus did produce a more granular picture of arbitration outcomes than FINRA’s binary non-zero recovery measure, particularly in identifying the presence of multi-outcome cases. We also found, however, that while FINRA makes public far more information than other arbitration forums, that information is insufficiently standardized, limiting measurement and analysis. In the Sections that follow, we present our results, and also a set of recommendations that would enable a more sophisticated understanding of outcomes and even better definitions of success. We describe the results of the following text analytics approaches: (a) targeted term frequencies and keyness measures to identify win/loss/settlement language within the awards; (b) sentiment analysis to identify positive versus negative language in the “Award” segments of the documents; and (c) targeted token extraction to extract all dollar amounts in the “Relief Requested” and “Award” segments, to measure the ratio of amount requested to the amount awarded.
A. Targeted Term Frequencies and Keyness Measures
As noted previously, as a pre-processing step before further analysis, we segmented each arbitration decision into its component parts, listed in Figure 3 above. To identify win/loss/settlement language within each award, we isolated the sections marked as “Award” (AWD) and “Other Issues Considered and Decided” (OICAD). A manual review of the decisions confirmed that the arbitrators consistently reported their resolution of the claimants’ claims in these sections. Figure 4 below is a word cloud showing the most frequent 100 terms in the combined AWD and OICAD sections for all 3,227 awards, where larger font size indicates higher term frequency.151 While word clouds are not very analytically useful, they do provide an impressionistic sense of the kinds and frequency of words that dominate a passage of text.
The problem that is immediately evident from this word cloud is that the most frequent terms within the combined AWD and OICAD sections are generic to the arbitration process: finra, respondent, panel, hearing, claimant, arbitrator, resolution. The term “expungement” is also quite common, and refers to a respondent’s request, routinely made along with defenses, to have their regulatory record wiped of the claims against them.152
Digging down, some much smaller words and phrases do suggest an outcome: dismiss, denied, shall pay. However, these meaningful substantive words appear to be relatively rare among the top one hundred terms. This suggests that unsupervised classification and clustering analytics techniques may be inappropriate for the task of outcome identification. These techniques “learn” from latent differences within the text and then classify, or cluster, like documents with like documents.153 However, the differences in our corpus between documents that contain “win” words and documents that contain “loss” or “settlement” words appear to be quite subtle, given those substantive words’ relative infrequency in the corpus. This type of text structure can thwart machine learning algorithms’ classification and clustering abilities.
Given this text structure, we adopted a supervised approach, guided by human subject matter expertise. We first read 439 arbitration awards (14%), chosen at random from our corpus of 3,227. We manually classified those awards as containing a claimant win, loss, settlement, or any combination of the three. We also identified lists of key words and phrases in the AWD and OICAD sections that signaled the claims’ outcome(s). Those full lists appear in Table 1 in Appendix A, along with measures of the keyness, or centrality, of the words that distinguish each outcome category from the others.
We then parsed the text into sentence-level units and ran targeted term frequency counts to generate tallies of those words’ and phrases’ appearance, per sentence, in each document’s combined AWD and OICAD sections.154 We found the sentence-level analysis to be more effective than generating counts from the entire blocks of AWD/OICAD text because we could better identify the particular word forms and usages that indicated each win/loss/settlement outcome. We could also assign multiple outcomes to a single case, with win language coming from one sentence in the AWD section, for example, and loss language coming from another. From our term frequency tallies, we then generated counts of win/loss/settlement outcomes per arbitration decision and compared those against our own 439 hand-coded decisions and against FINRA’s reported summary statistics as validation and robustness checks.
1. Results Indicative of Customer Success
Our analysis revealed that most FINRA awards, almost eighty percent, could be classified by a single outcome of either a win, a loss, or a settlement of the claim. The remaining approximately twenty percent of awards exhibited characteristics of more than one, and sometimes all three, potential outcomes. Accordingly, we did not obtain a binary outcome driven purely by win (monetary recovery or equitable relief) or loss (denial of all claims), and instead, the textual analysis indicated that the outcomes were more nuanced. Figure 5 below illustrates the results of the textual analysis classifying all cases, by year, as single- or multiple-outcome cases.155
Among the single-outcome awards, language indicative of a settlement-only outcome outpaced language indicative of a win or a loss, as illustrated by Figure 6, below.
The dominance of the settlement outcome was not a surprising result, as FINRA’s own statistics report that, from 2015 to the end of 2019, roughly 53% of filed claims were resolved via direct settlement by the parties and another 12% were settled in mediation.156 What is surprising is the percentage of settlement-only awards contained in the data set: as illustrated by Figure 6 above, about 41% of single outcome cases were settlements. A settlement of a FINRA claim is typically documented privately between the parties with the underlying arbitration being dismissed upon execution of a satisfactory settlement agreement.157 That award documents solely related to settlement (41% of total single-outcome awards) were more prevalent than either complete wins (27% of single-outcome awards) or complete losses (33% of single-outcome awards) suggests that expungement, far from the “extraordinary” relief it is intended to be,158 is sought in a significant percentage of the customer cases in which a consumer settles the underlying claim.159
The percentage of single-outcome cases that our analysis coded as solely a customer win, 27%, is not comparable to the overall rate reported by FINRA for awards in which a customer obtains any monetary award for the same time period, 41%.160 However, when we broaden out from the single-outcome cases to include the remaining 20% of cases with multiple outcomes, our win rate is quite close to that of FINRA. As Figure 7 below shows, 38% of cases in our data set included a win (regardless of other outcomes also present in the case), whereas FINRA reported non-zero damage recoveries for customers in average of 42% of cases, 2013–2018.161 These figures cannot be compared quite as directly, however, as our data set spans May 2013 to May 2018, whereas FINRA’s spans January to January. Nevertheless, the proximity of the two figures to one another supports the validity of our text-based method of identifying wins.
Nevertheless, our results call to mind the question first raised in the Introduction: Which outcomes should count as claimant wins? Our analysis elucidated a characteristic of customer recovery in FINRA awards not obvious from FINRA’s reporting of the binary recovery/no recovery metric: that in many instances, investors who recovered some monetary award also settled or lost part of their claim(s).162 The multiple-outcome awards metric identified by our research thus provides an additional layer through which consumer outcomes in FINRA proceedings can be evaluated.
For instance, as Figure 8 below shows, 49% of the multiple-outcome cases suggested both a win and a loss (10% of all awards), perhaps indicating the success of a subset of claimants against all respondents or indicating the success of all claimants against a subset of respondents in a proceeding.
Further, a win and a settlement were present in 28% of the multiple-outcome awards, or 5% of all awards, suggesting, perhaps, a claimant or claimants with particularly strong claims. A loss and a settlement together were present in 16% of the multiple-outcome cases (3% of total awards), perhaps suggesting a claimant willing to take a risk after obtaining some monetary relief in a settlement. Finally, a very small percentage of the multiple-outcome cases, 7% (or 1% of all awards), exhibited characteristics of a win, a loss, and a settlement. Taken together, the measurement of multiple outcomes provides a different frame through which to view the results of a FINRA arbitration proceeding and illustrates the complexities that may not be apparent from a review of the aggregate, binary claimant success statistics reported publicly by FINRA.163
2. Validation and Reliability of Results
In addition to comparing our “win” results to FINRA’s non-zero recovery totals, as described above, we further validated our results by comparing them to the 439-case sample of manually classified outcomes.164
In 79% of the hand-classified awards, the machine and expert coding of documents as containing wins, losses, and/or settlements were an exact match. Of the remaining 21%, one-third were multiple-outcome cases that the automated process correctly identified as multiple-outcome, but incorrectly identified the particular constellation of outcomes at issue. The remaining incorrectly-classified outcomes were approximately evenly distributed among misidentified wins, losses, and settlements.
Future work will continue to fine-tune this text analytic approach to outcome identification. Appendix B, for instance, contains the results of a keyness analysis of the three categories of single-outcome case: wins, losses, and settlements.165 There, we ran code to calculate the prevalence of every word across the AWD/OICAD corpus. Assuming an even distribution of words across the corpus, this prevalence measure allowed us to generate an expected frequency for each term within each arbitration decision’s block of AWD/OICAD text. We could then compare each term’s actual frequency in each decision—its observed value—to its expected frequency. Those terms with observed values in a particular block of text that exceeded their expected value were more “key” to that text and performed better at distinguishing that block of text from others.166 The resulting word lists are included here in Appendix B; we are working to integrate these results into our outcome identification measures in future iterations of this research.
In conclusion, our targeted term frequency approach to identifying language indicative of wins, losses, and/or settlements generated results capable of validation by comparison to FINRA percentage results and an expert review. We expect performance to improve in future extensions of this research. Even this first attempt at outcome classification shows levels of nuance that may better describe the complexity of FINRA arbitration awards and taxonomy of claimant outcomes in a fashion that is not currently captured by FINRA.
B. Sentiment Analysis
In addition to the keyword approach described above, we experimented with another text analytics technique known as sentiment analysis, which uses expert-assembled dictionaries of positive and negative words to assign a polarity score to a block of text.167 We used a set of sentiment word lists developed by communications scholar Lori Young and computational political scientist Stuart Soroka known as the Lexicoder Sentiment Dictionary.168 The dictionary contains 2,858 negative sentiment word patterns and 1,709 positive sentiment word patterns.169
Using this method, we were able to assign a positive or negative score to the AWD segment text in 3,216, or 99.7%, of the documents in our corpus.170 We counted text as positive if it scored more than 50% positive on the Young and Soroka measure.
To summarize our results, shown below in Figures 9 and 10 for all cases and single-outcome cases, respectively, we found some differences in polarity among arbitration decisions that we had classified as containing wins, losses, and settlements, but the difference was not substantial. In particular, while losses contained more negative AWD language than wins, settlements and wins were approximately equally positive. This conclusion held even in single-outcome cases, where we might expect the wins to contain substantially more positive language than the losses and settlements.
We also refined our analysis, investigating the distribution of positive language scores across outcome types, rather than merely looking above and below a 50% positivity cutoff. Here again, however, the positivity gaps among the three outcome types were not substantial enough to generate a reliable outcome classification on the basis of sentiment scores alone. The three panels in Figure 11 illustrate these results. Each panel shows a kernel density plot of the proportion-positive scores for the AWD text for single-outcome arbitration decisions identified as wins, losses, and settlements. The height of the plot line on the y axis indicates the number of awards with the proportion-positive score shown on the x axis.
Here, the win and loss plots each peak at around 50% positivity, though the win plot is generally more skewed to the right, indicating that more awards contained higher proportions of positive language. Unhelpfully, however, the settlement plot is even more right-skewed, indicating even more heavily positive language.
Thus, we conclude that sentiment analysis, standing alone, is an insufficient identifier of an award’s outcome, failing to provide a meaningful measure of whether a customer-claimant prevailed or lost in an arbitration proceeding.171
C. Targeted Dollar Amount Extraction to Measure the Ratio of Amount Awarded to Amount Requested
The Introduction posed the question of whether a claimant’s recovery of 68% of claimed damages and another’s recovery of 0.0005% should count equally as wins. We posit that the answer to this question is “no.” Instead, we suggest that the ratio of compensatory damages awarded to an investor’s claimed losses would provide a more refined assessment of whether a customer prevailed in a particular matter, especially when combined with additional information about the claimant’s position throughout the arbitration proceedings.172
This approach would improve upon FINRA’s current outcome reporting scheme where only monetary recovery or lack of monetary recovery is reported. FINRA does not claim that non-zero monetary recovery constitutes a win, though lay interpretation of these undescribed measures likely leads to such an assumption. The recovery of only a small portion of the amount requested might not feel like a customer victory on any level other than pure principle, and its inclusion with other non-zero monetary recoveries should perhaps not be included—expressly or by implication—as a win.173 Instead, such cases where a respondent was able to significantly reduce its liability exposure by taking the claim through to a final hearing and award might be better classified as a consumer loss and an industry win.
We thus attempted to build a text analytics tool to extract all dollar amounts from the Relief Requested (RR) and AWD segments of each arbitration award. Specifically, we attempted to isolate amounts of compensatory damages from all RR sections and compare them to the equivalent compensatory damages awards within all AWD sections.
We did so by isolating the dollar sign symbol within the text, extracting the words in small windows before and after, and searching for the phrase “compensatory damages” within those windows.174 Applying this methodology, we identified 1,459, or 45% of the awards in our corpus, with a dollar amount in both the RR and AWD segments, in close proximity to the phrase “compensatory damages.”
At first glance, this result seems promising, as it is relatively close to FINRA’s average non-zero recovery rate over the years 2013–2018 of 42%.175 However, the authors’ close review of the RR and AWD sections of the approximately 400 hand-coded awards revealed problems with this text analytics approach to calculating a request-award ratio—and generated some recommendations to improve data access. The remainder of this Part addresses the problems; the Part that follows offers our recommendations.
1. The Prevalence of Unspecified and Non-Monetary (Equitable) Damage Requests Hinders Measurement of Percentage of Relief Requested that Is Subsequently Recovered
Relying on a comparison of raw numbers does not account for the significant number of awards in which a customer seeks damages but elects not to specify them in her statement of claim or primarily seeks damages that are not reduceable to a monetary sum. For example, FINRA allows consumers to list damages as unspecified or to be proven at a hearing.176 Here, 11% of awards had no dollar amount listed in the RR section. Of those awards, the term “unspecified” appears in 42%—almost half.177 A claimant’s reasons for requesting unspecified relief cannot be determined from evaluation of the award itself, and while a complete denial of the claim would be indicative of a complete loss,178 the award of any dollar amount does not necessarily indicate a win because there is nothing to which it can be compared as a measure of success.
Moreover, a claimant may seek equitable relief that may not be expressed as a dollar figure, such as specific performance179 or rescission.180 For example, the terms “rescind,” “rescission,” “specific performance,” and “equitable” appear in 16% of awards in which there is no dollar amount present in the RR section, and when requested as the primary remedy, such proceedings may result in an award of equitable relief or a monetary award if the claim is not settled or denied in its entirety.181
Alternatively, an RR segment that does include a dollar figure might result only in equitable relief that provides the consumer complete relief in a non-monetary fashion.182 The 11% of proceedings in which no dollar amount was identified in the RR segment of an award may possibly be explained by a claimant who seeks unspecified damages or equitable relief.
Similarly, a dollar amount was contained only within the AWD segment, but not in the corresponding RR segment, in about 4% of awards. This may be explained by an initial request for unspecified damages or equitable relief that the arbitrators, after a hearing, subsequently reduced to a sum certain of monetary damages. Thus, if the claimant seeks unspecified relief or seeks or is awarded equitable relief, a measure of the percentage of amount recovered from the initial amount requested is infeasible.
2. In Multiple-Party Cases, the Presentation of Amounts Requested and Amounts Recovered Is Not Uniform, Rendering Comparison Impracticable
Second, the means by which dollar amounts are presented in the RR and AWD sections of a FINRA award may not be uniform, making it impossible to determine whether individual claimants obtained relief in the amount requested or in the amount they believed due to them from a particular respondent. For example, in many FINRA arbitration proceedings, multiple claimants seek relief against one or more respondents or one claimant seeks relief against more than one respondent. Such proceedings often include an aggregate sum of the compensatory damages sought in the RR section.183
This would not be a problem if the AWD recorded compensatory damages in the same lump sum format. However, compensatory damages in the AWD segment may be expressed in many different formats, including, but not limited to: (1) amounts that a single respondent is ordered to pay to each claimant, listed by claimant,184 which may include a successful outcome for one claimant but not another;185 or (2) amounts that multiple respondents are required to pay, due to joint and several liability, to each claimant, listed by claimant.186
These problems are compounded in proceedings that are consolidated or severed. FINRA’s dispute resolution forum provides liberal joinder rules as well as provisions for severing parties from proceedings.187 FINRA’s Code of Arbitration Procedure also provides for the consolidation of claims.188 In arbitrations where proceedings are consolidated or a party is severed from the initial case, the measurement of amount awarded is much more difficult to determine if the initial compensatory damages was an aggregate amount. As one example, in an award where two cases were consolidated, the RR section listed aggregated amounts sought in each of the underlying cases by FINRA proceeding number but awarded damages listed by claimant without regard to which of the two underlying cases and aggregate damages requests to which each individual claimant initially belonged.189
Similarly, awards in proceedings that were severed and decided in separate actions make it impossible to determine the claimants’ overall percentage of monetary success when the amount sought in RR is aggregated across all respondents.190 Accordingly, in multiple-claimant and multiple-respondent cases, it can be impossible to construct a ratio of damages recovered to damages claimed in a fashion that accurately reflects each claimant’s success.
3. The Presence of Amended and Supplemental Relief Requested Throughout the Lifecyle of a Proceeding Renders Measurement of a Single Percentage of Amount Recovered Difficult
The RR segment of an award often contains multiple dollar figures representing the amount requested by the claimant or claimants at varying points along the lifecycle of the proceeding. A claimant, in many cases, may seek an initial amount of compensatory damages (or unspecified damages) in the statement of claim and then file an amended statement of claim seeking a different measure of damages,191 and/or seeking a different measure of damages at the conclusion of the hearing.192 Determining which, if any, of these sums should be used even if they are capable of comparison to the amount awarded is difficult, if not impossible, because the reasons for the claimant’s decision to change the amount requested is not contained within the award document.193
4. Claimants Often Request a Range of Damages Reflecting Different Damages Theories Instead of a Sum Certain
Related to the prior point, a claimant may initially choose to express her damages against respondent(s) in a range194 when the law permits her to seek different measures of damages or she seeks damages in the alternative.195 From a range expressed in aggregate against multiple respondents, unless the proceeding ends with no settlement and the claimant recovers nothing against any respondent, it is otherwise impossible to determine the extent to which a claimant succeeded.196 Measuring outcome as an expression of the percentage of amount recovered from a range of the amount requested does not capture the complexities present in such multiple-party and multiple-outcome cases and may either overstate or understate success.197 Nor can a range be practically used to measure success in a single-claimant/single-respondent proceeding because the measures of damages anchoring the low and high ends of the range may both be recoverable. Accordingly, the practice of using ranges in the amount requested complicated our efforts to test this measure.
5. Relief Requested Includes a High Incidence of Multiple Dollar Amounts, Making Machine Classification of Total Relief Requested Less Reliable
Fifth, the frequency of multiple dollar amounts in the RR segment of an award makes it difficult to determine, without a manual, expert review of each document, the total amount of compensatory damages sought by a claimant or claimants.198 Though some dollar amount was found in 3,035 of the proceedings (94%), the number of unique dollar amounts was 11,283, or 3.5 times the number of awards in which any dollar amount was found. While a machine is capable of adding these sums together, in many cases, doing so would likely dramatically overstate the actual damages to which the claimant was entitled.
For example, punitive damages, while recoverable in a FINRA proceeding,199 are often reduced to a dollar amount in the RR segment,200 but they are not a measure of the actual damages suffered by the claimant and, if not granted, significantly impact the percentage measure of a claimant’s recovery.201
6. Proceedings in Which Underlying Claims Are Settled but Result in a Final Award on an Associated Person’s Request for Expungement Cannot Be Expressed as a Percentage of Amount Awarded of Amount Requested
Actions seeking expungement after the underlying claims were settled significantly impacted our ability to reliably analyze the percentage recovered in an award as a measure of success.202 On the one hand, the granting of an expungement would indicate that a customer did not succeed in the proceeding because an expungement is only permitted to be granted when the claim “is factually impossible or clearly erroneous,”203 the associated person was “not involved” in the conduct underlying the claim,204 or “the claim, allegation or information is false.”205 Thus, given such a high bar that must be met to obtain an expungement, if expungement is granted as relief in a customer arbitration award in which the underlying claims were dismissed or settled, one could argue that the measure of damages awarded the customer is $0, resulting in 0% recovered of the amount requested.206 On the other hand, a proceeding that the consumer agreed to settle, if they obtained some financial recovery, might be characterized as recovering greater than 0% of the amount requested in the proceeding.207 These differing potential interpretations caused us to further investigate proceedings that were held open solely to rule on an associated person’s expungement to determine whether we could devise a metric to ascribe an amount of award (≥$0) in such cases.
As an initial matter, we looked to other work investigating expungement proceedings. Other research has shown that expungements are granted in an extremely high percentage of FINRA proceedings in which a customer settles the underlying claim.208 One study, for example, found that in settled customer cases that remained open solely for the purpose of a hearing on an associated person’s expungement request, expungement was granted nine out of ten times.209 Such research has raised significant questions and concerns regarding whether arbitrators are granting expungements in proceedings where they ought to have denied the request,210 leading FINRA to provide additional guidance to arbitrators on the “extraordinary” nature of expungement and the high bar that must be met before it is granted.211
This research then prompted us to investigate how frequently the so-called “extraordinary” relief of expungement was sought in customer cases. We started by reviewing the textual analysis describing consumer proceedings with a single outcome as a settlement-only proceeding.212 Of the 2,599 proceedings in which a single outcome was recorded, a complete settlement was identified in 41%, or 1,060 proceedings. This result indicates that in 33% of the full data set, an award was issued for the sole reason of ruling on a broker’s expungement request. In our targeted term frequency analysis of results, the word “expungement” appeared with a greater frequency than any word associated with the results of a consumer case. Accordingly, the word “expungement” is one of the largest in the word cloud in Figure 4 above.
A term associated with expungement existed in 1,961 out of the 3,227 awards, or 61% of total awards. A phrase associated with expungement appeared at a much more significant frequency in awards with language suggesting a settlement or dismissal of a claim. In proceedings in which the textual analysis indicated that the award reflected only a settlement, a phrase associated with expungement appeared in 1,040 out of 1,060 cases, or 98% of the awards. In proceedings in which the textual analysis indicated that there was some indication of settlement but the presence of some other outcome, a phrase associated with expungement appeared in 55% of the awards.
As a result of these challenges, expungement proceedings following a total settlement should be excluded from this measure because they artificially lower the claimant’s outcome when no settlement sum is available and may not reflect the consumer’s perception of recovery in the rare instance when a dollar amount of settlement or percentage of amount recovered is recounted in the expungement award.
7. Partial Settlement of Claims and Respondents Who Fail to Appear to Defend May Affect the Reliability of the Proposed Measure of Success
Relatedly, in cases where the textual analysis indicated some aspect of settlement along with a win or loss and a dollar amount was present in the AWD section, it is often impossible to determine whether the percentage of amount recovered to amount requested is an accurate reflection of how the case concluded. In such cases, there is no indication of the amount of the settlement obtained and/or its distribution among parties and claims.213 In a few cases, while the panel noted that the amount awarded was adjusted to reflect the settlement obtained by the claimant, the panel did not reveal the actual settlement amount.214 The lack of information related to an underlying settlement may make subsequent awards less reliable because a lower percentage of recovery of the initial amount requested may not be indicative of the full recovery obtained by the claimant, thus understating the measure of success.
On the other hand, the prevalence of proceedings in which a claimant obtained a settlement with some respondents and elected to proceed to award against non-appearing respondents might artificially inflate the percentage awarded the claimant. Given the challenges arising with claims that proceed to an award after a partial settlement, whether the respondent is represented or not, we do not believe that any sum recovered post-settlement can be reliably compared to the amount requested without knowledge of the underlying settlement amount and relative culpability of the respondents.
8. Awards Where Claimant Obtains Monetary Recovery and Where None of the Respondents Appear or Defend May Not be Reliable
Though they are a small number of overall FINRA awards, FINRA has recognized that some awards claimants receive in FINRA proceedings go unpaid.215 FINRA has the ability to enforce arbitration awards against members and the stockbrokers they employ by conditioning continued membership in FINRA upon payment of arbitration awards.216 When, however, a respondent is no longer registered with FINRA, FINRA’s ability to mandate recovery diminishes, and a significant percentage of the unpaid awards come from cases in which a respondent did not appear.217 Moreover, it may be argued that even though FINRA arbitrators are not permitted to condition an award on the non-appearance of a party,218 when a party does not appear, the claimant has a better chance of recovering or supporting a more ambitious damages calculation because there is no one present to argue for a different damages measure.219
On the other hand, an argument can be made that a respondent may elect not to appear in a proceeding when the conduct is particularly egregious but where she is no longer a FINRA member and does not believe the claimant will be able to collect, indicating that awards in such cases might be higher. These variables lead us to believe that the inclusion of any measure of amount awarded out of amount requested is unreliable at best when the award is against a party who does not appear to defend at a hearing.
In conclusion, a measure of the percentage that a consumer receives of the amount sought, though perhaps a more realistic measure of success than identifying proceedings in which any sum was awarded, proved to be unworkable, given the current structure of FINRA arbitration awards. Many of the awards suffered from one or more of the above-identified issues, indicating that any result we obtained from application of this measure would be incomplete and unreliable.
IV. Data Access Recommendations
Our computational text analysis suggests several recommendations that FINRA can enact to improve data access, from making currently available data more susceptible to independent research to providing additional information within award documents that is available elsewhere in the FINRA regulatory sphere.
A. Provide Access to Customer/Industry/Expungement Award Classifications
First, FINRA should designate and make available to researchers the more granular means by which it classifies awards. These classifications are used in FINRA’s published summary statistics, and so are evidently in use internally in FINRA’s own record-keeping procedures. However, they are not included in the metadata that accompanies individual arbitration awards available for download from FINRA’s online awards database. Making these classifications available for public searches of the awards database would substantially increase transparency and aid in analyses such as the ones conducted in this Article.
For example, the online awards database currently allows users to search for results only by panel composition, forum, and document type.220 The addition of a field classifying an award as deriving from a customer, expungement, or industry proceeding within these search features and their related metadata will better permit researchers to develop a corpus for study. Indeed, it is not currently possible to easily recreate the set of awards that FINRA includes in its binary measure of outcomes in “Customer Claimant Arbitration Award Cases”221 because one must first identify customer cases from within the universe of cases by delving—as we did—into the text of each award. Further, the text that describes each case’s “Nature of the Claim,” e.g., “Customer(s) vs. Member(s)/Associated Person(s)” is varied, requiring substantial hand classification, especially when a proceeding is held solely for an expungement after settlement of the underlying claims. Classifying the award documents in FINRA’s online database via the metrics that FINRA itself uses to track arbitration outcomes would promote better access, permitting researchers to independently assess FINRA’s conclusions.
B. Report Amounts Requested and Amounts Awarded in Same Format to Facilitate Measurement
As previously discussed, we propose a measure of claimant success that would account for the ratio of the amount awarded to the amount requested, plus additional information about the claimants’ position throughout the arbitration proceedings.222 Accordingly, we recommend that FINRA take steps to make this measurement possible within its forum. This recommendation, seemingly simple, has several component parts.
First, FINRA should create a section of the RR and AWD segments of an award document to list the total compensatory damages requested and awarded, respectively, per claimant and per respondent. Each section should reflect a set dollar sum, which may include $0. In the event a claimant initially seeks unspecified damages, FINRA should record the amount later requested at hearing. In the event such a case settles, FINRA should request the settling claimant(s) to report the amount of compensatory damages requested representing a fair assessment of the previously unspecified amount. Coupled with the recommendation in Section C, below, this would permit development of a success measure currently incapable of measurement.
In addition, we recommend that FINRA capture, in all cases, the last sum of compensatory damages sought by the claimant, even if it did not change from the statement of claim.223 For those cases that primarily seek non-monetary, equitable relief, the RR and AWD segments should either be reduced to the monetary equivalent of the requested relief, or both should reflect the request for (or declination of) equitable relief. Finally, FINRA should specially report whether an award was rendered for a claimant after a properly served respondent declined to appear and defend. Each of these recommendations will assist in the measurement of an outcome that may better reflect a claimant’s relative success or failure in the forum.
C. The Results of all Customer Claims, Whether Resolved via Settlement or Hearing, and the Amount Recovered Should be Reported in an Award Document
Finally, we recommend that FINRA’s awards detail the resolution of all customer cases, whether determined after a hearing or after the parties’ settlement, and include the amount the customer recovered. Such a recommendation is in line with other consumer arbitration forums and FINRA’s regulatory objectives. For example, in the AAA forum, AAA reports, on a per-claim basis, whether a claim is resolved via settlement, an award, or dismissal in a summary document that also lists the same information FINRA reports in its awards, such as the case number, amount of the claim, forum fees, representation of the parties, and arbitrators, among other things.224 In its role as a regulator and the licensing authority for broker-dealers and their associated persons, FINRA should go further than AAA, requiring the issuance of an award detailing the monetary recovery a customer obtains whether the case resolves via settlement or after a hearing.
Though reporting the results (and amount) of a private agreement to resolve a case sounds drastic, the resolution of customer claims—including the dollar amounts that change hands—is already available from FINRA in another public data set. As part of its regulatory function, FINRA maintains a database known as the Central Registration Depository (CRD).225 CRD is FINRA’s response to the federal securities law’s requirement that all securities exchanges “establish and maintain a system for collecting and retaining registration information.”226 An individual broker’s CRD record details information the broker provides when registering, changing firms, or after certain reportable events arise.227
Among the information that a broker is required to disclose is information concerning consumer complaints (whether or not they subsequently evolve into an arbitration proceeding) and the settlement of consumer claims above a threshold dollar amount.228 While primarily a regulatory tool, CRD also serves a public protection and education function.229 As a result, a subset of the information contained in an individual or firm’s CRD record230 is available through the public-facing online BrokerCheck tool. FINRA and other regulators actively encourage investors to consult the data provided by BrokerCheck when researching whether to engage a particular broker for advice.231 Accordingly, the settlement of a customer-initiated claim, as well as the amount of the settlement, is listed on the stockbroker’s publicly available BrokerCheck report, whether or not they were individually named in the underlying proceeding.232
This settlement information, which is already available to FINRA, and to the public via BrokerCheck, should be recorded via an award document available in FINRA’s online awards database.233 Because settlement information, unless it is expunged, is required to be reported on BrokerCheck, FINRA should go beyond AAA and issue awards that reflect settlement as a resolution and detail the amount of the settlement. Providing the results of all customer engagements in the FINRA forum in a single data repository would eliminate the selection bias we noted in the currently available corpus of awards234 in addition to aligning with FINRA-mandated disclosure requirements to its members and associated persons.
Conclusion
This Article is the first step in evaluating the consumer experience in FINRA arbitration and assessing whether the arbitration process lives up to its investor protection goals. As illustrated by the stories of Neil Harrison’s investor clients highlighted in the Introduction, choosing a frame with which to measure an investor’s engagement with the FINRA forum as a success or failure necessarily colors subsequent evaluations of the forum’s efficacy in compensating investors, deterring bad-actor brokers, and ensuring the integrity of the securities markets.
While computational text analysis provides tools with which to evaluate FINRA awards, only one tool is currently capable of measuring outcomes in the FINRA forum: targeted term frequencies were able to identify language within the awards and describe their outcome(s) as a win, loss, settlement, or some element of all three. We thus discovered that a binary classification of an award as a monetary recovery or zero monetary recovery does not adequately capture the complexities or nuance of FINRA proceedings, and recognizing that awards may contain characteristics of a win, a loss, a settlement, or a combination of the three singular outcomes may better and more reliably describe the operation of the FINRA forum.
Moreover, while sentiment analysis and a comparison of the amounts requested and recovered did not produce reliable measures of customer success, the failure of these methods and explanation of their limitations suggest discrete, accomplishable improvements to FINRA’s data collection and reporting procedures. We hope that our results and associated recommendations prompt FINRA and other arbitral forums to reassess how they collect and report information intended for public consumption and study.
In sum, we discovered that neither uniformly presented outcomes nor the existence of a large quantity of data necessarily produces information that is capable of measurement and classification through different textual analysis methods. This is because substantial complexity and variation may be masked by surface-level uniformity of presentation.
Moreover, our results suggest that the provision of data in an effort to promote transparency does not necessarily equate to actual transparency. Actual transparency would require data and documents that report useful, accurate, granular statistics and are organized in a fashion that is susceptible to outside research. The need for actual transparency is particularly pressing where, as here, FINRA’s dispute resolution forum is designed to lend trust to the securities markets.235
If FINRA were to implement our proposals, it would increase and standardize the information reported per each award and aggregate all of the available data concerning customer disputes for study, whether derived from FINRA’s regulatory, investor protection, or dispute resolution functions. Compiled as a single data set, these data would provide the best opportunity for researchers to determine how investors like Neil Harrison’s clients actually fare in disputes with broker-dealers. In the meantime, our research will continue to apply the tools of data analytics to FINRA’s dispersed public data and document sets as we assess the efficacy of FINRA’s investor protection measures and continue to pry open the arbitral black box.236
Appendix A
Our targeted words and phrases are listed below, along with the rules we applied for exclusion and inclusion. Note that we accounted for misspellings and problem characters introduced into the text during the process of converting the original .pdf documents to machine-readable text format. For example, in addition to “settlement,” we counted all appearances of the following, which we had previously identified as variants of “settlement”: “settlementâ,” “settlementsâ,” and “settlernent.”
Appendix B
