How to Use Secondary Research Surveys To Manage Your Firm: A Primer

 

By Dr. John W. Olmstead

Copyright 1996-2002 by Olmstead & Associates. All Rights Reserved.

The 21st century is presenting law firms with new challenges. The general business economy is in turmoil and law firms are facing new risks and uncertainties. Clients are no longer tolerating arrogance and mediocre services. Clients are holding law firms to higher service standards. In order to prosper in the 21st century, law firms are going to have to drastically change their models for conducting business. Organizational performance, effectiveness, and leadership must rise to higher standards. General management, problem solving, and action taking skills must be enhanced. Firms will have to improve their overall marketing initiatives. This will require that many firms improve their overall management effectiveness and use every management tool available. Law firms will need to identify “best management practices” that can be employed to enhance management effectiveness.

The Search For Best Practices

Managing partners, executive directors, legal administrators, marketing directors, and practice group leaders express common concerns regarding the lack of benchmark data and general information on “best practices.” Law firm management needs answers to questions such as:

• How are we doing financially?
• Is our attorney/staff ratio in line with comparable firms?
• Should and how do we offer new practice areas?
• Are our billing rates competitive with other firms?
• Is our partner compensation in line with comparable firms?
• Are we paying competitive salaries to our management and staff professionals?
• How do we maximize client loyalty and the amount of business from each client?
• Should and how do we open a new office?
• What lawyers should we recruit as lateral hires?
• How do we position the firm in the market to most effectively compete for desired business?
• Which marketing tactics will be the most effective?

Information on best practices can come from either primary research conducted or commissioned by the firm or secondary research. Primary research is often employed in the form of analysis of internal practice data, client surveys, positioning studies, and sometimes to support strategic planning efforts. However, firm management often wants to know what other firms are doing for “benchmarking” purposes and uses secondary survey research for this purpose.

Numerous organizations offer “off-the-shelf” varieties of secondary research surveys. In the area of law firm management, Altman & Weil’s Survey of Law Firm Economics and PricewaterhouseCoopers’ surveys are noteworthy and have been used by law firm management for years as their primary source of benchmark data. Both of these surveys provide detailed benchmark data in areas such as:

• Billable hours
• Billing rates
• Compensation
• Firm specialization
• Personnel ratios
• Financial performance

Other law firm management consulting firms offer similar surveys. General management benchmark surveys are available from general management consulting firms and general management professional associations. Surveys on specific topics of interest such as law firm financial performance, salaries and compensation, and marketing budgets are also available from national and local bar associations and law firm management and marketing professional associations such as the Association of Legal Administrators and the Legal Marketing Association. The Legal Marketing Association noteworthy “2001 Survey of Law Firm Marketing Budgets” was published this year.

Research Crash Course

Prior to jumping in and reviewing that survey report, you might want to brush up on research fundamentals. Just as the legal, management, and marketing fields have their own rules, procedures, and vocabulary, so does the field of professional research. Standards and ethical guidelines are specified by professional research associations such as Qualitative Research Consultants Association, American Marketing Association, and the Market Research Association. The Legal Marketing Association has published research guidelines for commissioning and conducting market research.

Neither time nor the scope of this article permits a comprehensive treatment of the fundamentals of sound research. Thus, the following key points will provide an outline of important concepts:

KEY POINT #1:What is the purpose of the research? The most common purposes of research are exploration, description, and explanation. The purpose of the original research, as well as your purpose if using secondary research, will determine appropriate research methodology and how existing research should be used. Exploratory studies often use qualitative research designs. Typically, when the purpose of the research involves comprehensive description and explanation, research designs are quantitative-descriptive designs that require either census studies of the entire population using descriptive statistics or random or quota samples using both descriptive and inferential statistics. In such cases the objective is to be able to make inferences and statements about the larger population of interest. In other words, what can you say about a population based upon the results observed in a random or quota sample?

KEY POINT #2: Know the difference between qualitative and quantitative research and appropriate uses for each. Qualitative methods do not use quantity measurement; but rather attempt to measure the “quality” of something. Examples include case studies and focus groups. Quantitative research attempt to measure “quantity.” Quantitative studies would include statistical market surveys, budget surveys, salary surveys, and financial analyses. Quantitative research is the backbone of clinical research and they yield more “scientific” findings than qualitative studies. Qualitative studies are often used as initial exploratory studies and are subsequently supplemented by quantitative studies.

KEY POINT #3: The user of a research study must understand the difference between a population, census and a sample. A research or study population is the larger group of interest while the sample is the group we draw to study. An example of a population would be all law firms in the United States, while a sample would be a subset of the population that is representative of the overall population. In other words a sample is a set of respondents selected from a larger population for the purpose of the survey. A census is a one-by-one count of the entire population.

KEY POINT #4: Do not generalize or make inferences beyond your data. Know the population of the study and whether a census or representative sample was taken. Limit generalizations to the population of the study or to the specific sample if extensive bias or error exist in sample data.

KEY POINT #5: Keep in mind that sample surveys yield accurate results when the following kinds of errors are avoided:

• Coverage Error
  Coverage error occurs when the list – or sampling frame – from which a sample is drawn does not include all elements of the population you wish to study. This type of error can be reduced by compiling the best list or sampling frame.
• Sampling Error
  Sampling error occurs when we only survey a subset or sample of all people in the population instead of conducting a census. This type of error cannot be eliminated but it can be reduced by insuring that the sample is random and large enough.
• Measurement Error
  Measurement error occurs when a respondent’s answer to a given question is inaccurate, imprecise, or cannot be compared in any useful way to another respondent’s answers.
• Nonresponse Error
  Nonresponse error occurs when a significant number of people in the survey sample do not respond to the questionnaire and are different from those who do in a way that impacts the accuracy of the study. Low response rates serve as a warning that nonresponse error might be a problem. Depending on who is surveyed and what method is used, anything under 60-70 percent should be a red flag. Telephone surveys yield much higher response rates than mailed surveys.

KEY POINT #6: Have a working knowledge of the following statistical concepts:

• Descriptive vs Inferential Statistics
  Descriptive statistics uses data for descriptive purposes and not for making predictions. Thus, descriptive statistics consists of methods and procedures for presenting and summarizing data. The methods most commonly employed in descriptive statistics are the use of tables and graphs, and the computation of measures of central tendency and variability such as the mode, median, mean, range, percentiles, variance, and standard deviation. Measures of association or correlation are also categorized as descriptive statistical procedures. Inferential statistics employs data in order to draw inferences (i.e., derive conclusions) or make predictions. Typically, inferential statistics sample data are employed to draw inferences about one or more populations from which the samples have been derived. Inferential statistics primarily employs sample data in two ways to draw inferences about one or more populations. The two methodologies employed in inferential statistics are hypothesis testing and estimation of population parameters. Results are typically expressed in terms of statistical significance levels and confidence intervals respectively.
• Continuous or Categorical Data
  Continuous data consists of data which are comprised of quantity values. Examples include: age, number of years of employment, distances, test scores, yearly income, etc. Categorical data consists of data which are, very simply, forms of data which fall into groupings or divisions. Examples include: gender (male/female), political affiliation (Republican/Democrat/Independent/other), and favorite color (blue/green/orange/yellow). The use of either continuous or categorical data has a profound effect on statistical analysis methods available for use. A quick review of the relationships between these data types is mandatory. First, however, note that information collected as continuous data can later be categorized to form categorical data. However, information originally collected as categorical data can never later be made continuous. As an example, age data originally collected in four age ranges (categories) could never be re-evaluated later as continuous data, however, continuous age data can easily be categorized into “age groups”. Thus, continuous data is always more statistically “robust” or powerful. When treating data as continuous, more exact information is always yielded. Accurate assessments of mean, range, standard deviation, variance, and other statistics also become possible. If data are originally collected as categorical, much detail is forever lost, and no precise assessments for mean or median are ever possible.

  The key point to remember here is that if data are of the categorical form, values for the mean, median, mode, and all other descriptive statistics are nonsensical. Instead, count and percentages should be reported, without reference to a mean, median, etc.

• Measurement Scale
  Data falls into one of four measurement scales, going from the weakest measurement scale (nominal) to the strongest measurement scale (ratio): nominal, ordinal, interval, or ratio. Nominal data, the lowest measurement scale, simply uses numbers (or any symbol) to identify objects, people, places, etc. Nominal data has no quantity measure. Nominal data is always categorical. Ordinal data, simply sets numbers into some rank order. The order can be established to the highest to lowest or the reverse. An example would be a baseball team’s place in the standings or a ranking of top performing stocks. Ordinal data is usually considered categorical and has order, but not quantity. Interval data, indicates a quantity without regard to a true zero-point. The best example of interval data is temperature and IQ scores. Interval data is always continuous. Ratio data, the most powerful measurement scale, provides a quantity value with regard to a true zero-point. Examples include: income, fees collected, age, weight, height, and percentage test scores. Ratio data is always continuous.

  Remember: Nominal and ordinal data are categorical. Interval and ratio data are continuous. Use continuous data whenever possible. Use counts and percentages for categorical data and other statistics for continuous data.

KEY POINT #7: Know the meaning and the strengths and weaknesses of the mean, median, mode, variance, standard deviation, and variance. Know the dispersion of your data and how it is skewed. When the distribution of the data is mound shaped (bell-shaped curve) the mode, median, and mean are all equal. The mean is always more sensitive to extreme values and when such is the case the distribution will be more skewed (not a normal bell shaped curve). In skewed distributions, the median is usually a more accurate indicator.

KEY POINT #8: Know the characteristics of your data and whether the report is using the right statistics and graphs to describe the data.

KEY POINT #9: Know when a study is so flawed that it is totally worthless.

KEY POINT #10: Have a plan on how the research will be used to assist the firm in formulating management action initiatives.

Tips For Using Secondary Research Surveys

Secondary research surveys must be used with caution. Many of the surveys that are available as off-the-shelf products fail to meet research guidelines and standards. Some are census studies with extremely low response rates. Others are broader national surveys in which samples were used. However, most often these samples were not constructed as random/quota samples and in many cases the sample consisted of clients, prospects, and contacts of the sponsoring organization as opposed to a representative sample of the study population. Many of these samples have been entirely respondent self-select studies. Consequently, many of these studies are potentially biased and contain various degrees of error.

Follow these tips both prior to purchasing and when using secondary research surveys.

TIP #1: Know your objectives and how you will use the research. Is it exploratory in nature and for basic benchmarking or is it intended for comprehensive description or explanation? This will help you determine which studies will satisfy your needs, how rigorous your standards should be, and how to use and interpret these studies.

TIP #2: Before purchasing secondary research surveys learn all that you can about the details of the study. Know the limitations of the study. Ascertain the following:

• Qualitative vs quantitative design
• Census or a sample
• Particulars on the study population and sample
• Details on how sample was constructed, pulled, size, etc.
• Details on response rates and any other information on the four types of error
• Questionnaire used
• Statistics used and appropriateness
• Characteristics of the data
• Overall soundness and quality of the survey and the organization sponsoring the survey

TIP #3: If you purchase a survey keep Tips 1 and 2, as well as Points 1-10, in mind as you review and interpret the data contained in the survey.

TIP #4: Look for results that matter.

TIP #5: Look for information that you need to know to initiate changes in your management practices and marketing initiatives.

TIP #6: Review the questionnaire to determine how the questions were initially constructed.

TIP #7: Insure that the right statistics are being used to describe the data. (continuous vs categorical data)

TIP #8: Do the findings make sense?

TIP #9: Identify red-flag areas of concern where your firm may need to investigate further.

TIP #10: Formulate a plan for following up on areas requiring additional investigation and possible action.

The use of sound secondary research surveys can be invaluable and can assist firms in their quest for “best practices.” While law firms should strive to use surveys that meet the test of sound research, this is not always possible since no other source of information may be available. In other words – some information may be better than no information at all. In such situations law firms may decide to use research surveys that do not satisfy sound research guidelines. Such information can still be useful for exploratory analysis and when the information will be used for “benchmark” purposes. However, it is important for the firm to keep in mind the limitations of the study.

Many of the national law firm management surveys are designed to provide the information necessary for law firm management to evaluate their firm’s performance relative to comparable law firms. Statistics included in these studies represent broad performance benchmarks against which an individual firm can be measured. Law firms can use this information to compare their firm’s performance with other firms as a whole, as well as with firms of similar size, geographic location, population, practice specialty, etc. Keep in mind that the objective of such comparisons is to identify potential “red flags” that warrant additional investigation. Deviations between your firm’s figures (for any performance measure) and figures in the survey is not necessarily good or bad. It merely runs up the red flag which alerts you investigate further. Information in these surveys should be used as guidelines rather than absolute standards.

It is important that surveys used for exploratory or benchmark purposes not be taken as the gospel. They may not be representative or they may be contaminated with many of the errors mentioned previously. Finally, valid benchmark statistics may not be appropriate goals for your firm. For example, while several national surveys have reported that 2.4% of revenue is the median budget for marketing investments in large law firms, this may not be the appropriate goal. Large CPA firms are spending closer to 10%.

I hope this primer helps you on your journey.

John W. Olmstead, Jr., MBA, Ph.D, CMC, is a Certified Management Consultant and the president of Olmstead & Associates, Legal Management Consultants, based in St. Louis, Missouri. The firm provides practice management, marketing, and technology consulting services to law and other professional service firms to help change and reinvent their practices. Founded in 1984, Olmstead & Associates serves clients across the United States ranging in size from 100 professionals to firms with solo practitioners. Dr. Olmstead is the Editor-in-Chief of “The Lawyers Competitive Edge: The Journal of Law Office Economics and Management,” published by West Group. He also serves as a member of the Legal Marketing Association (LMA) Research Committee. Dr. Olmstead may be contacted via e-mail at jolmstead@olmsteadassoc.com. Additional articles and information is available at the firm’s web site: www.olmsteadassoc.com

Copyright © 1996 - 2024 Olmstead & Associates, All Rights Reserved

Home – Resource Center – News Releases – About Olmstead – Services Provided – Contacting Olmstead – Sitemap