Gender Issues: Analysis of Methodologies in Reports to the Secretaries of
Defense and Army (Letter Report, 03/16/98, GAO/NSIAD-98-125).
GAO reviewed three studies on gender-related issues affecting initial
entry training in the Department of Defense (DOD), focusing on: (1) how
the groups conducted their work; (2) how well the work supported making
conclusions and recommendations; (3) the availability of documentation
supporting the report; and (4) the extent to which the final report
described the study methodology and disclosed limitations.
GAO noted that: (1) the Army's Senior Review Panel on Sexual harassment
used four methods to collect data: individual interviews, focus groups,
surveys, and observations; (2) during its 8 months of work, the panel
visited 59 installations worldwide, conducted interviews with 808
military and civilian Army personnel, ran focus groups with over 8,000
soldiers and civilians, and surveyed 22,952 individuals; (3) the use of
multiple methods of data gathering, the rigor with which the various
methods were conducted, and the publication of the data in the report
provides ample support for making conclusions and recommendations; (4)
the Federal Advisory Committee (FAC) on Gender-Integrated Training and
Related Issues used focus groups as its primary method of data
gathering; (5) although FAC conducted over 300 focus groups and
individual interviews, their value for making conclusions and
recommendations is limited because the Committee did not: (a)
systematically collect the same information from all groups; (b)
document the information generated in each of the interviews and focus
groups; or (c) explain how what was heard in the interviews and focus
groups led to their conclusions and recommendations; (6) in addition,
the length of the focus group sessions, the number of participants, and
the number of questions addressed may not have provided adequate time
for full participation of the respondents on all issues; (7) given these
limitations, the extent to which the Committee's work supports its
conclusions and recommendations cannot be determined; (8) the Defense
Advisory Committee on Women in the Services (DACOWITS) also used focus
groups of trainees, trainers, and supervisors in the Army, Air Force,
Navy, Marine Corps, and Coast Guard to identify what issues concerned
women and men at training installations; (9) members of the DACOWITS
held focus group discussions at 12 schools at 9 installations in the
United States and prepared a summary report of the results at each
installation; (10) the DACOWITS Chair used these to prepare a report to
the Secretary of Defense that accurately reflected the opinions and
perceptions cited in the individual installation reports; (11) the
DACOWITS focus groups were: (a) larger than recommended in the
literature; (b) were sometimes not long enough to allow meaningful
participation; and (c) were not recorded or documented on a
group-by-group basis; and (12) the DACOWITS report summarized the
opinion and perception data obtained from focus groups; and (13) it made
no conclusions or recommendations on military training based on that
information.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-98-125
TITLE: Gender Issues: Analysis of Methodologies in Reports to the
Secretaries of Defense and Army
DATE: 03/16/98
SUBJECT: Military training
Sexual harassment
Military personnel
Military facilities
Women
Data collection
Advisory committees
Surveys
Evaluation methods
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
March 1998
GENDER ISSUES - ANALYSIS OF
METHODOLOGIES IN REPORTS TO THE
SECRETARIES OF DEFENSE AND THE
ARMY
GAO/NSIAD-98-125
Gender Issues
(703235)
Abbreviations
=============================================================== ABBREV
DACOWITS - Defense Advisory Committee on Women in the Services
DOD - Department of Defense
Letter
=============================================================== LETTER
B-279460
March 16, 1998
The Honorable Charles S. Robb
Ranking Minority Member
Subcommittee on Readiness
Committee on Armed Services
United States Senate
The Honorable Ike Skelton
Ranking Minority Member
Committee on National Security
House of Representatives
The Honorable Martin T. Meehan
House of Representatives
Over the past year, several groups have examined gender-related
issues affecting initial entry training in the Department of Defense
(DOD). The studies that have gotten the most attention are the
Army's Senior Review Panel's report on sexual harassment, the report
of the Federal Advisory Committee on Gender-Integrated Training and
Related Issues, and the report of the Defense Advisory Committee on
Women in the Services. Because the groups produced somewhat
different results, our objectives were to (1) describe how the groups
conducted their work, (2) determine how well the work supported
making conclusions and recommendations, (3) assess the availability
of documentation supporting the report, and (4) determine the extent
to which the final report described the study methodology and
disclosed limitations.
Since each of the three studies used focus groups as either their
primary data gathering method or in concert with other means of
collecting information, we used social science literature on focus
groups to assess their use of that methodology. Focus groups use a
small group discussion format, guided by a moderator, to gather
information on a given topic. Focus groups are useful in providing
an understanding of the depth and the variety of opinions, feelings,
or beliefs, but are not useful for quantifying the extent to which a
particular view or opinion is held. For these reasons, focus groups
are a qualitative technique best used when the aim is to explore an
issue or, at the other end of the research process, to assess
reaction to specific proposals. As with any methodology, its value
is dependent upon the extent to which it is appropriate to the
objectives of the inquiry, how well the methodology is conducted, and
the way in which the results are used. More information on focus
group methodology is found in appendix I.
BACKGROUND
------------------------------------------------------------ Letter :1
The reports of sexual misconduct at Aberdeen Proving Grounds led the
Secretary of Defense to establish the Federal Advisory Committee on
Gender-Integrated Training and Related Issues and to ask the Defense
Advisory Committee on Women in the Services to meet with trainees and
trainers. These incidents also prompted the Secretary of the Army to
establish the Senior Review Panel on Sexual Harassment.
In November 1996, the Secretary of the Army established the Senior
Review Panel on Sexual Harassment. The panel's mission was to make
recommendations to improve the human relations environment in which
soldiers live and work, with the specific goal of eradicating sexual
harassment in the Army. The panel consisted of seven members,
including two retired general officers recalled to active duty, two
active duty general officers, a senior noncommissioned officer, and
two DOD civilians. The Senior Review Panel forwarded its report\1
and recommendations to the Secretary of the Army in July 1997. It
included 40 recommendations, of which 14 dealt with training and
related issues.
In June 1997, the Secretary of Defense established the Federal
Advisory Committee on Gender-Integrated Training and Related Issues.
The Secretary of Defense established the Committee as a result of the
sexual misconduct incidents at Aberdeen Proving Grounds. Former
Senator Nancy Kassebaum Baker chaired a panel of 11 that included
civilians, retired officers, and a retired senior noncommissioned
officer. The Secretary directed the Committee to assess the training
programs and policies of the Army, Navy, Air Force, and Marine Corps
and make recommendations to improve initial entry training. The
Committee issued its report\2 to the Secretary of Defense on December
16, 1997. It made 30 recommendations covering the full cycle from
recruitment through basic and advanced training.
The Defense Advisory Committee on Women in the Services (DACOWITS)
has been advising secretaries of Defense since George Marshall
established the Committee in 1951. DACOWITS, which consists of 30 to
40 civilians, makes recommendations to the Secretary on the roles of
women in the Armed Forces and on quality of life issues affecting
readiness. As part of its mission, DACOWITS members conduct annual
visits to selected Army, Air Force, Navy, Marine Corps, and Coast
Guard installations, both here and overseas. These visits serve two
purposes: (1) to provide the Secretary of Defense with insight into
the thoughts and perceptions of servicemembers in the fleet and the
field and (2) to determine what issues DACOWITS will concentrate on
in the future. In November 1996, the former Secretary of Defense
requested that DACOWITS visit training installations to meet with
trainees and trainers in the training environment. In February 1997,
the current Secretary of Defense endorsed the request. DACOWITS
provided a report\3 to the Secretary of Defense summarizing these
visits. In its report, DACOWITS recommended continued visits to
training installations, but made no recommendations on military
training.
--------------------
\1 An Army summary of the findings of this report appears in appendix
II.
\2 A DOD summary of the report's key recommendations and findings
appears in appendix II.
\3 A summary of the findings excerpted from the report's executive
summary can be found in
appendix II.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
The three studies have different objectives and were conducted
somewhat differently. We compared the methodologies of the three
studies with the principles of focus group methodologies set forth in
social science literature. We found a wide variance among the three
studies. The Army's Senior Review Panel's study most closely
followed the methodology for conducting focus groups recommended by
the literature. Specifically, we found:
-- The Army's Senior Review Panel on Sexual Harassment used four
methods to collect data: individual interviews, focus groups,
surveys, and observations. During its 8 months of work, the
panel visited 59 installations worldwide, conducted interviews
with 808 military and civilian Army personnel, ran focus groups
with over 8,000 soldiers and civilians, and surveyed 22,952
individuals. The use of multiple methods of data gathering, the
rigor with which the various methods were conducted, and the
publication of the data in the report provides ample support for
making conclusions and recommendations.
-- The Federal Advisory Committee on Gender-Integrated Training and
Related Issues used focus groups as its primary method of data
gathering. Although the FAC conducted over 300 focus groups and
individual interviews, their value for making conclusions and
recommendations is limited because the Committee did not (1)
systematically collect the same information from all groups, (2)
document the information generated in each of the interviews and
focus groups, or (3) explain how what was heard in the
interviews and focus groups led to their conclusions and
recommendations. In addition, the length of the focus group
sessions, the number of participants, and the number of
questions addressed may not have provided adequate time for full
participation of the respondents on all issues. Given these
limitations, the extent to which the Committee's work supports
its conclusions and recommendations cannot be determined.
-- The Defense Advisory Committee on Women in the Services also
used focus groups of trainees, trainers, and supervisors in the
Army, Air Force, Navy, Marine Corps, and Coast Guard to identify
what issues concerned women and men at training installations.
Members of DACOWITS held focus group discussions at 12 schools
at 9 installations in the United States and prepared a summary
report of the results at each installation. The DACOWITS Chair
used these to prepare a report to the Secretary of Defense that
accurately reflected the opinions and perceptions cited in the
individual installation reports. We noted that the DACOWITS
focus groups were (1) larger than recommended in the literature,
(2) were sometimes not long enough to allow meaningful
participation, and (3) were not recorded or documented on a
group-by-group basis. The DACOWITS report summarized the
opinion and perception data obtained from focus groups. It made
no conclusions or recommendations on military training based on
that information.
THE ARMY'S SENIOR REVIEW PANEL
ON SEXUAL HARASSMENT
------------------------------------------------------------ Letter :3
The Army's Senior Review Panel on Sexual Harassment formed four
teams, one to review Army policies and three for data collection.
Each field team consisted, on the average, of six military personnel
and one civilian. The Chair, the Vice-Chair, or the Deputy Assistant
Secretary of the Army (a member of the panel) accompanied each field
team during their visits. Other panel members traveled with the
teams as often as possible.
Visits lasted 1 to 4 days depending on the numbers of participants in
the various activities. Before the visits, the participants for the
individual interviews, focus groups, and survey were selected and
scheduled. Generally, the visit started with a briefing to present
the purpose of the activity and a description of the team's data
collection efforts. Next, the team divided into smaller groups to
conduct individual interviews, conduct focus groups, or administer
surveys. These activities ran concurrently and team members rotated
to different activities at different times. Visits ended with a
briefing providing commanders the opportunity to begin corrective
actions.
Panel members and the working group collaborated in writing the
panel's report. Once a near final draft was generated, the panel
members met for a final review and agreement on the content. The
panel on Sexual Harassment issued its report to the Secretary of the
Army in July 1997.
METHODOLOGY
---------------------------------------------------------- Letter :3.1
The Army's Senior Review Panel on Sexual Harassment used four methods
to collect data: individual interviews, focus groups, surveys, and
observations. According to the senior social scientist detailed to
the panel, the field teams conducted interviews and focus groups
using carefully developed protocols to obtain human relations
environment information. Members of field teams conducted individual
interviews with 808 military and civilian Army leaders and personnel
in Army support groups. Focus groups consisted of randomly selected,
single-gender groups of 8 to 12 people organized by ranks or
categories.\4 Participants totaled 7,401 soldiers and 1,007
civilians. Facilitators and note takers of the same gender as the
groups conducted the sessions. All data obtained through these two
activities were entered into a computer for analysis. The working
group, which consisted of more than 40 military and civilian
personnel, developed main themes or categories and placed the
perceptions data under the categories. Data were then analyzed by
rank, by gender, and by question.
The written surveys addressed leadership, cohesion, and sexual
harassment. Field teams administered the surveys to 22,952
servicemembers. Surveys were developed for trainees, trainers, and
the general Army population. The working group analyzed survey data
using a standard statistical analysis software package.
Statistically projectable results appear in the report by question
and, in some cases, by gender. Observations were made during visits
to barracks and other facilities and to engage in informal
conversations with military and civilian personnel, family members,
and others.
The seven panel members, supported by the working group, collected
data at 59 Army installations worldwide selected using a stratified
random sampling design. Stratification was based on the type and
location of the installation. The study took 8 months to complete
and obtained information from over 32,000 Army personnel.
--------------------
\4 Rank or category included, junior enlisted soldiers, junior
noncommissioned officers, senior noncommissioned officers, company
grade officers, field grade officers, civilian employees, drill
sergeants, instructors, and trainees.
USE OF MULTIPLE
METHODOLOGIES AND EXTENSIVE
DOCUMENTATION ALLOWED MAKING
CONCLUSIONS AND
RECOMMENDATIONS
---------------------------------------------------------- Letter :3.2
The panel's methodology supported making conclusions and
recommendations. Focus groups were used in conjunction with surveys
to not only confirm the survey data but also provide texture and
perspective to the data. The focus groups were of an appropriate
size and were all asked the same questions, in the same order by
trained moderators. However, the number of questions asked of many
of the focus groups was significantly greater than the five or six
questions recommended by focus group literature. For example, the
set of questions for trainee focus groups consisted of 15 questions
and the set for trainers consisted of 13 questions. Focus group
discussions were not tape recorded because it was feared that this
would inhibit the participants, but notes were taken by a note taker
and were content-analyzed. The notes from each focus group session
were destroyed, after the responses were entered in the database and
verified for accuracy, to assure that participant confidentiality was
maintained. Destroying the original documentation to assure
confidentiality is considered an appropriate measure by social
scientists. In addition, the completed survey forms were also
destroyed to assure participant confidentiality.
In volume two, the panel provides an extensive discussion of its
methodology. Volume two provides details on how participants were
selected, copies of the focus group questions, the surveys, and the
interview questions. Results of the surveys were included in the
report as well as the most frequently heard responses in the focus
groups. Furthermore, a sufficient amount of data are presented in
volume one of the report, which outlines the panel's conclusions and
recommendations, to allow the reader to evaluate them.
LACK OF SURVEY PRETEST
RESULTS IN CONTROVERSY
-------------------------------------------------------- Letter :3.2.1
An area of controversy arose because the survey developers did not
pretest the survey questions. We were told by the senior social
scientist attached to the panel, that tight time frames precluded the
panel from carrying out a pretest of the survey form. Normally, a
pretest is performed to identify problem questions, problems with
language interpretations, unclear instructions, or to determine if
there are some questions that respondents will refuse to answer. In
this instance, the survey form contained six questions that some
respondents in early administrations found inflammatory, offensive,
and an invasion of privacy and resulted in some refusals to complete
the survey. Subsequently, those questions were eliminated and a
revised form was used. The data on the six questions were not
included in the database, which resulted in an accusation that the
panel had eliminated important data from its analysis. The report
disclosed the problem and its resolution in the methodology section.
We believe that the panel acted responsibly in eliminating the
offending questions to avoid having a negative effect on the survey
return rate. The controversy, however, demonstrates the importance
of pretesting survey forms before conducting a survey.
THE FEDERAL ADVISORY COMMITTEE
ON GENDER-INTEGRATED TRAINING
AND RELATED ISSUES
------------------------------------------------------------ Letter :4
The Federal Advisory Committee on Gender-Integrated Training and
Related Issues saw its role as listening to the views of trainees,
trainers, supervisors, and service officials and providing the
Secretary of Defense with its best judgment about what should be done
to improve training. Small teams of Committee members visited 17
training installations and operational units to gather opinions.
Most Committee members visited installations from two services.
While the Committee Chairman visited installations for all four
services, no Committee member or Committee staff member visited all
of the installations.
Once at an installation, the Committee members followed the same
general schedule: reveille, breakfast with new servicemembers,
meetings with command officials, and interviews and focus groups
before lunch. After lunch with support personnel, the Committee
members conducted additional focus groups and interviews. At the end
of each visit, they met with command officials to discuss their
findings. Installation visits generally lasted 1 day, although
visits to basic training sites were 2-day trips. The visits to the
training installations and operational units occurred in September
and October 1997.
The Committee had two public meetings, the first in July 1997 and the
second in October 1997. At the July meeting, service representatives
provided information on the services' recruiting and training
programs. At the October 1997 meeting, Committee members discussed
their observations and agreed to a partial list of recommendations
for the report. The Committee's staff drafted the report based on
the discussions they heard during their installation visits and the
public meeting, and memorandums submitted by some of the Committee
members in preparation for the October meeting. Committee members
received the draft report in early December and revisions were made
based on their comments. The Committee Chair discussed the report
with Committee members in a series of one-on-one telephone calls to
arrive at the final recommendations.
METHODOLOGY
---------------------------------------------------------- Letter :4.1
The Committee's primary means of collecting information involved
focus group discussions. The Committee held 199 focus groups,
soliciting opinions from more than 1,000 trainees, 500 trainers, 300
first-term servicemembers, and 275 supervisors at U.S. training
installations and operational units. Participants were randomly
selected under the supervision of the installations' inspectors
general. Trainees who participated in the focus groups were within 2
weeks of completing their training. Participants in the trainer
focus groups were trainers for at least 1 year. First term
participants were in their initial assignment and had been on the job
between 6 and 18 months. Generally, the Committee met with equal
numbers of females and males, although, because of the limited number
of female trainers and supervisors, this was not always possible.
Focus groups included about 10 to 15 people each and were
gender-segregated. All of the focus groups were moderated by
Committee members, and generally two Committee members or a Committee
member and a Committee staff member attended each session. The
Committee members worked from a set of questions tailored for each
service and each type of focus group. Although the number of
questions varied by type of focus group, the set of questions for all
basic training focus groups consisted of 20 questions, some of which
had multiple parts. While some focus groups were scheduled to last
only 30 to 45 minutes, most focus group sessions lasted nearly an
hour.
Committee members also conducted over 100 interviews with service
officials, including commanding officers, inspectors general, company
or squadron commanders, and senior noncommissioned officers. They
also met with representatives of support groups such as chaplains,
equal opportunity officers, medical officers, and legal officers.
PROBLEMS WITH THE
METHODOLOGY LIMIT THE
USEFULNESS OF THE REPORT
---------------------------------------------------------- Letter :4.2
The value of the information included in the Committee's report for
making conclusions and recommendations is limited because the
Committee did not follow recommended focus group methodology. The
Committee believed that a more flexible approach to the discussions
would enhance the quality of the exchange between the participants
and the Committee members. However, the fact that the same questions
were not asked of each similar focus group, along with the number of
questions, size of the groups, and length of the sessions may have
combined to limit full discussion. In addition, the focus groups'
discussions were not systematically recorded. As a result, the
extent to which the recommendations are supported by the Committee's
work cannot be assessed.
The Committee staff provided the Committee members with questions for
the focus groups. However, according to the staff director, the
Committee members were told that the questions were guidelines and
that they did not have to be asked as written. Because the Committee
members had the flexibility to ask any question they desired, the
responses should not be compared with each other. Also, the number
of questions provided to the panel members were far more than the
five to six focus group literature recommends. For example, the
staff provided 20 questions, some of which had several parts, for the
Committee to ask Army trainees at Fort Leonard Wood, Missouri.
Fifteen trainees participated in the 1-hour focus groups at this
installation. If the entire hour was spent on the questions, there
would have only been 3 minutes spent on each question and only 15
seconds for each participant to respond. We do not believe that
would have been enough time for a meaningful discussion of a
question. Finally, even if all the questions were asked as they were
written, they were not always asked in the same order each time.
Social science literature suggests that the same questions asked in a
different sequence may result in different responses.
The absence of documentation of the comments made in the individual
focus groups was the most serious methodological shortcoming. While
the Committee members took notes during each focus group, these notes
were not made part of the Committee's records nor were they
summarized and included in the report. Without documentation, it is
impossible to determine if the Committee's work supports its
recommendations. Also, the lack of documentation prevented the
Committee from analyzing the data to know what comments they heard or
how often similar comments were made. Knowing how often a particular
kind of comment was made and the subgroup of the person who made it
are ways of putting the comments in perspective and filtering biases.
The report of the Federal Advisory Committee on Gender-Integrated
Training and Related Issues does not include a sufficient discussion
of the Committee's methodology and work process. For example, the
report states that the Committee conducted discussion groups with
randomly selected servicemembers, but it does not explain the random
selection process. In addition, the use of terminology such as
"randomly selected" implies a level of scientific rigor that was not
achieved in this study. The report does not identify the make-up of
the discussion groups, discuss what type of data analysis was done or
not done, or mention any limitations with the data. Limitations that
we believe should have been mentioned are that the report was based
on opinions and the results cannot be generalized to the entire
military training population.
Also, the report often presents opinions in a manner that they can be
misinterpreted as facts based on empirical data. For example, the
report says that the Committee members observed that integrated
housing is contributing to a higher rate of disciplinary problems,
but, according to the Chairman, the Committee did not obtain any data
to support this statement. In addition, the report contains many
statements that include words like "most", "many", and "majority".
These words lead a reader to believe that the Committee counted
responses to particular questions or polled the focus group
participants. The Chairman said that the Committee does not have
quantitative data.
The report also does not explain the process the Committee used to
formulate its recommendations. Although the Committee held a public
meeting in October 1997 after its installation visits had been
completed, the recommendations on separate barracks for male and
female recruits and on the organization of gender-segregated
platoons, divisions, and flights were not made until after that
meeting. Furthermore, those recommendations were not discussed by
the Committee as a whole, but rather in a series of calls to
individual Committee members.
THE DEFENSE ADVISORY COMMITTEE
ON WOMEN IN THE SERVICES
------------------------------------------------------------ Letter :5
The mission of the DACOWITS effort was to provide the Secretary of
Defense with an overview of broad issues raised by trainees and
trainers of both genders throughout initial entry training. A
secondary purpose was to help determine what issues DACOWITS would
concentrate on in the future. The Chair and the Executive Director
of DACOWITS selected seven members (all were women) to visit training
installations. Members were selected based on their DACOWITS
experience and the quantity and quality of their previous
installation reports. Typical visits were conducted by one DACOWITS
member and lasted 2 days. Visits began with a briefing by the
commanding officer about the school and its mission, followed by
trainee, trainer, and supervisor focus groups. At the end of a
visit, the DACOWITS member met with command officials to share the
results of the focus groups. Reports, summarizing the most
frequently heard comments from the various focus groups, were written
at the conclusion of each visit. In addition, the seven members met
at DACOWITS' 1997 fall conference to discuss the results of their
visits. Using the reports and the conference discussion, the 1997
DACOWITS Chair wrote the report. The report was released by the
Secretary of Defense in January 1998.
METHODOLOGY
---------------------------------------------------------- Letter :5.1
DACOWITS used focus groups as its primary means of data gathering.
Overall, they solicited the opinions of over 1,200 trainees,
trainers, and supervisors in the Army, the Navy, the Marines, the Air
Force, and the Coast Guard in focus group discussions at 12
gender-integrated training schools at 9 installations. The schools
included enlisted basic, intermediate, and advance training, and
officer advanced training. Most focus groups were gender-segregated
and trainees, trainers, and supervisors were in separate focus groups
as well. DACOWITS requested trainees with at least 40 percent of
training completed. Many trainees had completed their training and
were awaiting graduation.
The groups averaged 20 participants and sessions lasted about 60
minutes, although some were shorter. Before meeting with the
Committee members, focus group participants viewed an 18-minute video
that explained the mission of DACOWITS and highlighted some of the
gender equality issues that DACOWITS had worked on in the past such
as sexual harassment, discrimination, child care, and the combat
exclusion policy. The video set the stage for the two open-ended
questions that all the participants were asked: (1) "How is it
going?" and (2) "If you had five minutes to speak with the Secretary
of Defense, what would you tell him?" According to the former Chair,
DACOWITS uses these questions during all installation visits.
Training installation visits took place between July and November
1997.
At the conclusion of each visit, a DACOWITS member completed a
standardized installation visit report summarizing the most
frequently heard comments from the focus groups. The comments
included in these reports were entered into a computer and sorted by
frequency across the services as well as by individual service.
Issues were included in the report to the Secretary based on
frequency. The individual installation visit reports support the
opinions and perceptions that appear in the report to the Secretary
of Defense.
PROBLEMS WITH THE
METHODOLOGY LIMIT THE
USEFULNESS OF THE REPORT
---------------------------------------------------------- Letter :5.2
Some focus groups may have been too large or may not have had enough
time to allow ample participation by most of the participants. The
literature suggests that focus groups should be no larger than
12 participants. During the DACOWITS visits to the training schools,
some groups were as large as 20 participants. Groups larger than 12
usually do not allow sufficient opportunity to actively participate
in the discussion and are more difficult to manage. Also, the
majority of the sessions were about an hour long and some ran for
only 45 minutes, about half the time recommended by focus group
literature.
DACOWITS used two questions to generate discussion. However, time
may still have been a problem, since the questions were very
open-ended and could be taken in virtually any direction by a
participant. This would likely have the effect of increasing the
amount of time needed as each participant not only answered the
discussion questions, but also reacted and responded to the issues
raised by others.
DACOWITS did not document the individual focus groups as recommended
by focus group literature. Instead, DACOWITS members prepared
installation visit reports which summarized the opinions they heard
most frequently. While the installation reports document the work
performed and the issues surfaced during the training installation
visits, they do not capture enough information about the discussions
in each focus group to be really useful. For example, they do not
provide enough information on the rank or gender of the groups that
raised the issue which would help put the comments into perspective.
As we stated earlier, all of the DACOWITS members making installation
training installation visits were women. Some focus group literature
suggests that the gender of the moderator and the gender of the focus
group should be the same, particularly when the issues being
discussed are sensitive or have a direct bearing on the opposite sex.
Also, some focus group literature suggests that men are more likely
to tell a woman moderator what they think will impress or please her
rather than what they think. The use of female moderators for male
focus groups, in conjunction with the women's advocacy impression
that the video is likely to have conveyed, may have made some males
hesitant to raise issues or perceptions that might be construed as
anti-female. Because DACOWITS did not document each of its focus
groups it is impossible to determine if the use of women moderators
with all-male focus groups had an effect on the responses of the male
participants.
The DACOWITS report provides some methodological information for the
reader, but does not provide some key information. First, the report
does not provide any details on how the Committee members documented
the focus groups. Second, the report does not clearly explain the
process used by DACOWITS to determine what issues would be included
in the report. Third, while the report provides some detail about
the make-up of the focus groups it does not describe how the focus
group participants were selected. It should be noted however, as
recommended by focus group literature, the report clearly states its
two major limitations: (1) the opinion and perception information
included in the report has not been independently validated or
confirmed and ( 2) the Committee did not visit any gender-segregated
training facilities. Also, in accordance with the limitations of the
methodology, the DACOWITS report made no conclusions or
recommendations on military training.
AGENCY COMMENTS
------------------------------------------------------------ Letter :6
We provided a draft of this report to DOD, the Chairman of the
Federal Advisory Committee on Gender-Integrated Training and Related
Issues, and the former Chair and Military Director of DACOWITS for
comment.
We discussed our report with Department of the Army officials, who
concurred with our observations on the Army's Senior Review Panel on
Sexual Harassment. We also discussed the draft report with the
Executive Director of the Federal Advisory Committee on
Gender-Integrated Training and Related Issues who suggested some
clarifications to the report, which we considered and made as
appropriate. In addition, we discussed the draft with the military
director of DACOWITS, who stated that DACOWITS does not aim to meet
the standards of academic research but instead uses focus groups to
collect opinions and identify issues for further study. Finally, we
discussed the draft with the former Chair of DACOWITS who suggested
some technical corrections which we made as appropriate.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :7
We reviewed the reports from the Army's Senior Review Panel on Sexual
Harassment, the Federal Advisory Committee on Gender-Integrated
Training and Related Issues, and DACOWITS. We reviewed literature on
the conduct and use of focus groups, since that was a common
methodology across the three studies. We focused on the
methodological information provided in the reports, including any
limitations on the use of the information. We reviewed supporting
documents to determine if the evidence collected supports making
conclusions and recommendations. We did not evaluate the validity of
specific conclusions and recommendations made by any of the studies.
We met with the Chairman and Executive Director of the Federal
Advisory Committee on Gender-Integrated Training and Related Issues,
the former Chair and Military Director of DACOWITS, and with the
senior social scientist of the Army's Senior Review Panel on Sexual
Harassment to thoroughly explore the approach and methodology used in
these efforts.
Our review was requested by the former Ranking Minority Member of the
House National Security Committee and Mr. Meehan. We are addressing
the report to the current Ranking Minority Member of the House
National Security Committee, Mr. Skelton, as a courtesy. We are
addressing this letter to Senator Robb because it is related to other
work on gender issues in the military that we have undertaken at his
request.
We conducted our review in February and March 1998 in accordance with
generally accepted government auditing standards.
---------------------------------------------------------- Letter :7.1
We are sending copies of this report to the Secretaries of Defense,
the Army, the Air Force, and the Navy; the Chairman of the Joint
Chiefs of Staff; and the Commandant of the Marine Corps. We will
make copies available to any other interested parties.
The major contributors to this report were Carol R. Schuster,
William E. Beusse, Carole F. Coffey, George M. Delgado, and
Kathleen M. Joyce. If you or your staff have any questions
concerning this report, please call me on (202) 512-5140.
Mark E. Gebicke
Director, Military Operations
and Capabilities Issues
FOCUS GROUP METHODOLOGY
=========================================================== Appendix I
Focus groups\1 are carefully planned small group discussions
involving people with similar characteristics who are knowledgeable
about an issue but do not know each other well. The views expressed
in focus groups are not necessarily representative of a population
and statistical estimates cannot be derived from the results.
Furthermore, focus groups cannot be used to determine the extent of a
problem. Focus groups produce qualitative data that provide insights
into attitudes, perceptions, and opinions of the participants. They
are most often used before, during, or after quantitative research
procedures such as surveys. For example, focus groups can be used
before a survey is undertaken to help a research team learn about the
target audience or determine the appropriateness of the
questionnaire. Focus groups are often used with surveys to confirm
findings and to obtain greater breadth and depth of information.
Finally, focus groups are often used as a follow-up to surveys to
help interpret responses. On occasion, focus groups are used alone
when opinions and perceptions are more important than how many people
hold such views.
The size of the focus group is an important, but often overlooked,
element of a successful group discussion. The literature on focus
groups suggests that an appropriate size for a focus group is 6 to 12
people. A focus group with fewer than six participants sometimes has
problems with productivity because the group has fewer experiences to
share. Also, small groups can be more easily affected by people who
know each other, by uncooperative participants, or by participants
who view themselves as experts on the topic. Groups that have more
than 12 people usually do not allow people sufficient opportunity to
actively participate in the discussion, making the groups difficult
to manage.
The composition of the focus group is also important. Participants
should share some similar characteristics but be diverse enough to
allow for differences of opinions. The topic of discussion and the
information to be obtained dictate the types of characteristics
shared. However, generally participants should be similar in age,
occupation, education, and social class. Focus groups with distinct
differences among participants such as trainees and trainers or
junior and senior enlisted personnel do not work well because of
limited understanding of other lifestyles and situations.
Furthermore, some participants may be inhibited and defer to those
they believe to be better educated or more experienced or of a higher
social class. Sometimes, the gender of participants can affect the
outcome of a focus group and some social scientists recommend against
mixing genders because men and women tend to perform for each other.
When the opinions of disparate groups are needed, focus group
literature recommends holding separate groups for each distinct
group.
Focus group discussions are conducted informally and guided by
trained moderators who encourage participants to share their thoughts
and experiences. Trained, experienced moderators are critical to the
success of a focus group. An unqualified moderator can easily
undermine the reliability and validity of focus group findings.
Successful moderators are good listeners, who can make people feel
relaxed and anxious to talk. Moderators must control a group without
being obvious and be aware of time. Since literature suggests that
focus groups should be scheduled for 90 minutes and run no more than
120 minutes, moderators need to be able to keep the discussion on
track and move the participants from one topic to the next.
Moderators should be aware of the influence that they have on the
type and amount of data obtained. Moderators must be aware of their
own biases that might affect the validity of the data and take care
not to provide cues to participants about desirable responses and
answers. If dealing with sensitive subjects where views could vary
according to factors such as gender or race, it is recommended that
the moderator be similar in gender or race to the participants.
Finally, moderators must have sufficient knowledge of the topic to
put comments in perspective and followup on critical areas of
concern.
Questions are the heart of the focus group discussion. The
literature on focus groups suggests five or six questions for a
discussion group. The questions need to be carefully thought-out and
phrased to result in the maximum amount of information in the limited
time available. Questions should not suggest potential answers and
yes or no questions should be avoided. Questions should be asked in
the same order in every focus group and questions should be sequenced
from most important to least important to ensure that the most
necessary information is obtained from the participants if time runs
out. The sequence is important because the questions may interact
with one another to form the stimulus that generates the responses.
If the questions are asked in a different order at each focus group,
the stimulus is changed and the response will be different.
The results of the focus groups' discussions should be documented on
a session by session basis. Focus group literature agrees that the
best way to do that is by tape recording supplemented with written
notes. However, if a tape recording is not feasible or inhibiting to
the participants, note taking can be sufficient provided they are
complete enough to be analyzed.
A systematic analysis of focus group data is also important. The
analysis can be either qualitative or quantitative, but it must be
systematic and verifiable. It must be systematic in that it follows
a documented step-by-step process and verifiable to permit others to
arrive at similar conclusions using available documents and the raw
results. Social scientists have noted that there is a tendency for
novice researchers to see selectively only those parts of the
discussion that confirms their particular point of view. Often, a
researcher will go into the discussion with certain hunches of how
participants might feel. As a result, the researcher tends to look
for evidence to support these hunches and overlook data that present
different points of view. A systematic and verifiable process helps
researchers in filtering out bias and assuring that they present the
data as objectively as possible.
Once data are collected and analyzed, the data should be reported
and, if appropriate, conclusions and recommendations made. A report
should clearly state what the purpose of the study was, what its
scope was, how the data were collected and analyzed, and what, if
any, significant limitations exist on the data or the use of the
data. For example, studies that used focus groups as the primary
method of data collection should clearly state that the data being
reported is opinion or perception. If the opinions have been
substantiated by other types of data, this should be clearly stated
in the report. The report should also include the results of the
focus groups, and the results should be clearly stated so that a
reader can come to the same conclusions as the report writers.
--------------------
\1 This section drew from the following: Richard A. Krueger, Focus
Groups: A Practical Guide for Applied Research (Newbury Park,
California: Sage Publications,1988); David L. Morgan, Focus Groups
As Qualitative Research (Newbury Park, California: Sage
Publications,1988); and David W. Stewart and Prem N. Shamdasani,
Focus Groups: Theory and Practice (Newbury Park, California: Sage
Publications,1990)
SUMMARY OF REPORTS
========================================================== Appendix II
THE SECRETARY OF THE ARMY'S
SENIOR REVIEW PANEL ON
SEXUAL HARASSMENT
------------------------------------------------------ Appendix II:0.1
The following is excerpted from Army press reports that accompanied
the report of the Senior Review Panel's report on Sexual Harassment
in the Army as well as the executive summary of the report:
The panel found:
-- Sexual harassment exists throughout the Army, crossing gender,
rank, and racial lines; gender discrimination is more common
than sexual harassment.
-- Army leaders are the critical factor in creating, maintaining
and enforcing an environment of respect and dignity in the Army;
too many leaders have failed to gain the trust of their
soldiers.
-- The Army lacks institutional commitment to the Equal Opportunity
program and soldiers distrust the equal opportunity program.
-- Trainees believe the overwhelming majority of drill sergeants
and instructors perform competently and well, but "respect" as
an Army core value is not well institutionalized in the Initial
Entry training process.
Recommendations of the panel were broad-based and covered a wide
variety of Army processes including: leader development, equal
opportunity policy and procedures, initial entry training
soldierization, unit and institutional training, command climate, and
oversight.
FEDERAL ADVISORY COMMITTEE
ON GENDER-INTEGRATED
TRAINING AND RELATED ISSUES
------------------------------------------------------ Appendix II:0.2
The following is an excerpt from the press release that accompanied
the report of the Federal Advisory Committee on Gender-Integrated
Training and Related Issues which delivered its report to the
Secretary of Defense in December 1997:
"The panel studied the full training cycle including recruiting,
basic training, and advanced skills training. Its
recommendations covered the training cadre, housing of recruits,
fitness programs and follow-on advance training. Among the
several recommendations made for recruiting, the panel proposed
better preparing recruits mentally and physical for basic
training. It also recommended ways to improve the training
cadre. It recommended that physical training requirements be
toughened and made more uniform throughout the services. The
panel also suggested that emphasis on discipline be carried over
from basic to advance training. The panel recommended that
value training be incorporated into all initial entry training
programs and that training get more resources.
"During visits to training installations, the panel concluded
that men and women should be housed in separate barracks and
train separately at the operational unit level -- the Army
platoon, the Navy division and the Air Force flight. In the
Marine Corps men and women live, eat, and train separately. The
panel recommended that gender-integrated training continue for
field training, technical training and classroom work."
DEFENSE ADVISORY COMMITTEE
ON WOMEN IN THE SERVICES
------------------------------------------------------ Appendix II:0.3
The following is an extract from the executive summary of the report
to the Secretary of Defense from the Defense Advisory Committee on
Women in the Services:
"The scope of DACOWITS' training installation visits included
all elements of initial entry training, including basic
training, advanced individual training, and officer advanced
training. The majority of issues raised by trainees, trainers,
and supervisors of trainers were similar across all of the Armed
Forces.
"The most frequently raised issues by women and men and trainees
and trainers alike were artificial gender relationships imposed
at training installations, the persistence of gender
discriminatory behaviors at many locations; the relationship
between trainer attitudes and gender climates; the under
valuation of trainers, especially women trainers; the need for
greater gender integration to train field and fleet ready
servicemembers, the need to increase physical training
opportunities and standards; the need to improve screening of
new recruits and to harmonize recruiting quality and practices;
the under resourcing of training schools and the need to improve
support services for women trainees."
*** End of document. ***