When Shirley Malcom decided in 1963 to forego the University of Alabama, the still fitfully integrating school just 50 miles from her native Birmingham, and instead enroll at the University of Washington in Seattle, she says she ended up being the only Black person among 800 zoology majors. Eventually, Malcom says, another Black student saw her doing well academically and joined her. “He switched out of his major and came over to zoology,” she said. “Then there were two of us.”

Malcom graduated, earned a doctorate in ecology, taught high school biology, and eventually landed a post at the American Association for the Advancement of Science (AAAS), the national science advocacy group. There, she worked to boost representation for Black students and other underrepresented groups in science, technology, engineering, and math — now commonly referred to as the STEM disciplines.

By the end of the 20th century, her efforts, and those of countless others who had taken up the same cause, were paying off around the country. Black enrollment in engineering nearly tripled from 1970 to 1985, according to one report, and it continued to climb through the 1990s. The number of Black PhDs in engineering and the physical sciences were also rising. The needle, as Malcom and a coauthor would later write, was finally moving from “none” to “a few” — and there was cautious optimism that the trend would continue.

But that’s not what happened. An Undark analysis of nearly four decades of data on bachelor’s degrees awarded in the US suggests those hard-won gains for Black representation in the sciences are quietly slipping away, even as Black student representation in non-STEM fields has continued to grow. Culled from reports issued by the National Science Foundation (NSF), the analysis indicates that after decades of increases, the share of STEM-field bachelor’s degrees awarded to Black students peaked in the early 2000s and has been falling ever since — despite increasing federal spending on STEM diversity initiatives.

It is a critical reversal that has been largely overlooked, diversity advocates say, and they are troubled by the prospect that the growth in Black undergraduate STEM representation may have stalled. The numbers are “absolutely” concerning, said Sylvia James, NSF’s deputy assistant director of Education and Human Resources, “for me personally, and from the agency perspective.”

Precisely what is driving the decline is a matter of some debate. Some experts pointed to persistent income inequality and the disproportionate lack of access to quality schools among Black and other minority communities. Others argued that outreach efforts, peer mentoring, and other programmes aimed at fostering interest in the sciences among Black students have dwindled, causing enrollments to plummet. But several education and legal professionals also pointed to a more straightforward and sobering correlation: The steady downturn in STEM degrees among Black students, they say, comes in the wake of a large-scale retreat from specific programmes and policies that consider race in admissions, recruitment, and retention in higher education — policies commonly known as affirmative action.

Whether anything currently on the policy horizon can halt the downward trend remains unclear, but from her post at the AAAS, Malcom says she and her colleagues were noticing the potential fallout of this constellation of factors as far back as the late 1990s, when a wave of anti-affirmative action rulings were sweeping the country. They surveyed and visited dozens of STEM departments, gathering information on enrollment demographics and fellowship offerings. Even then, she said, the likely impact of social and political changes taking place at that time — and which remain key drivers today — seemed clear.

“We were able to go back to some of those same institutions that we had queried back in the late 80s, early 90s and show that, yes, we were in fact losing ground,” Malcom said. “Significantly losing ground.”

Undark’s analysis of STEM graduation rates, compiled from editions of NSF’s biennial report, “Women, Minorities, and Persons with Disabilities in Science and Engineering,” confirms one widely documented reality of US higher education: For as long as government agencies have been keeping track, there has existed a gap between Black Americans’ representation in STEM fields and their representation in the country’s general population. That representation gap, forged by institutional racism and perpetuated by socioeconomic disparities, remains vast to this day. But prior to the turn of the 21st century, it was narrowing.

In 1981, the earliest year for which reliable data were available, Black students received roughly 4.1% of all bachelor’s degrees awarded to US citizens and permanent residents in the life sciences, physical sciences, computer sciences, math, and engineering. By 2004, that share had climbed to 7.4%. (We represented Black students’ STEM degree completion rates as a fraction of STEM-field bachelor’s degrees awarded to US citizens and permanent residents to exclude effects of rising and falling foreign student enrollment; an NSF statistician consulted for this report said that the agency is currently transitioning to look at race and ethnicity data in the same way, calling it a “better way” to do the analysis. The social sciences, which have not traditionally been a central target of STEM diversity initiatives, were also excluded from this analysis.)

Since 2004, however, the proportion of STEM degrees awarded to Black students has been falling — even as the Black share of the US college-age population has held steady at around 14%. While the total number of Black STEM graduates did tick up over this period — from roughly 17,000 in 2004 to about 22,000 in 2016 — that expansion did not keep pace with the growth in STEM graduates overall.

As a result, the once-shrinking representation gap has begun to widen. In 2016, the most recent year for which NSF has published data, Black students received just 6.2% of US science and engineering bachelor’s degrees, down 16% from 2004 levels. Two experts who were contacted for this story — Karen Hamrick, a senior analyst at NSF’s National Center for Science and Engineering Statistics, and Catherine Weinberger, a labour economist at the University of California, Santa Barbara — conducted independent analyses and say they saw the same trend.

Although the declines are modest, they are likely consequential. Had Black students’ share of STEM-field bachelor’s degrees remained at its 2004 level, the US might have produced an additional 31,000 Black scientists and engineers over the 12-year span from then to 2016. Had the share continued to grow at the pace set in the 1990s, the additional STEM graduates might have been numbered close to 80,000.

While the NSF regularly publishes data on the race and ethnicity of STEM graduates, NSF media affairs specialist Michelle Negron confirms that the agency has not previously compiled comparable data over such a sweeping timeframe, saying that changes in definitions for race and ethnicity make it difficult to construct a consistent time series. Between 2008 and 2011, for example, surveys used to collect the NSF’s data transitioned to incorporate a “more than one race” category, which remains in use today.

But experts interviewed for this story say that, given the timing of that change, it is unlikely to explain the observed downturn in Black STEM representation. The agency has previously published data tracking STEM degree completion by race over the 20-year span between 1996 and 2016, and it regularly publishes race and ethnicity data spanning 10-year periods.

To the extent that the Black representation gap in STEM fields is widening, it is not for a lack of federal spending on diversity programmes. According to a report from the Office of Science and Technology Policy, in 2018 the US government spent $1.8 billion on programmes that seek to increase diversity, equity, and inclusion in STEM — more than half of the federal budget for STEM education programmes and more than double the sum spent on comparable programmes in 2008.

But legal and education professionals pointed out that, while funding for diversity programmes has continued to grow, funding for a particular class of diversity programmes has all but dried up: Federal agencies, universities, and private foundations have largely abandoned the use of programmes that are expressly limited to underrepresented racial or ethnic minorities.

Although a 1995 White House policy review noted that family origins, family affiliations with a school, and other advantages provided “countless scholarship programmes” that were, at least de facto, limited to White students, programmes aimed at minorities — often called race-targeted or minority-targeted programmes — only proliferated in the wake of the 1960s civil rights movement. By the 1990s, hundreds of scholarship, fellowship, internship, and mentoring programmes were courting minority students, especially African Americans, into the sciences and engineering. At one point, an estimated 5% of all undergraduate scholarships had minority status as a requirement.

Along with the integration of state universities in the South and a coordinated effort to establish science and engineering departments at Historically Black Colleges and Universities (HBCUs), these race-targeted recruitment and retention efforts were widely seen as key drivers of the late 20th-century surge in Black scientists and engineers. “No one denies the fact that we had the most rapid growth of minorities in STEM fields during this period,” wrote the authors of a 2011 National Academies report, referring to the era of race-targeted scholarships and fellowships.

As the 20th century rolled into the 21st and diversity programmes began to draw scrutiny from conservative political groups, however, that era would come to an end.

“I was able to see it. I saw it coming,” said Malcom, who around that time was serving as AAAS’s head of Education and Human Resources Programs. In a 1991 report, Malcom and colleagues at AAAS published results of a survey of hundreds of programmes aimed at recruiting women and minorities into the sciences and engineering, and they found that the efforts were fragmented, with little top-down coordination. “Everything was a little thing … and a lot of stuff was vulnerable,” Malcom recalls. “Five years later, a lot of those intervention programmes were exactly the things that were threatened.”

One of the first casualties of the campaign against affirmative action was a small scholarship programme for Black students at the University of Maryland, College Park. The Benjamin Banneker scholarship programme had been created in 1978 as part of a federally mandated plan to desegregate the school and redress past discrimination. In 1990, incoming freshman Daniel Podberesky, who is described in court documents as Hispanic, filed a discrimination lawsuit against the university after he was turned away from the programme. Four years later, a federal appeals court decided in his favour, holding that the university had not demonstrated sufficient evidence of lingering effects of past discrimination to justify the programme. When, in 1995, the US Supreme Court refused to rehear the case, it handed opponents of affirmative action a victory that would reshape the legal landscape for years.

Over the next decade, conservative advocacy groups would wage a relentless campaign against affirmative action in higher education. Lawsuits, settled out of court, led the NSF to abandon a minority graduate fellowship programme and jettison dozens of summer camps targeted at recruiting minorities into science and engineering. A summer science programme sponsored by the National Institutes of Health and the US Department of Agriculture also succumbed to legal pressure. A high-profile case against the University of Texas led the state of Texas to eliminate race and ethnicity outright as a factor in college admission, financial aid, and retention and recruitment. Although the Texas decision was later overruled, similar affirmative action bans were adopted in eight other states.

Through it all, the US Supreme Court has maintained that race-targeted programmes are, at least in some circumstances, constitutionally permissible as a remedy for past discrimination and as a tool to achieve a diverse student body. But even at institutions unbeholden to state affirmative action bans, the mere threat of a lawsuit was often reason enough to abandon race-targeted programmes, experts say. Federal agencies changed the names and eligibility language of diversity programmes and searched for race-neutral surrogates like family income or family educational history. Private foundations pulled funding from race-targeted scholarships and fellowships. Universities abandoned race-targeted summer training programmes, high school outreach programmes, college recruitment weekends, freshman orientations, and faculty-hiring programmes.

In 2004, Peter Schmidt reported in The Chronicle of Higher Education that nearly 70 universities, pressured by conservative advocacy groups, had either ended race-targeted programmes or opened them to students of all races. The retreat that had begun in earnest with the demise of the Banneker scholarship programme was, essentially, complete.

Mark Rosenbaum, a lawyer with Public Counsel, a pro bono public interest law firm based in California, says that the collective retreat from affirmative action — combined with the dismal state of K-12 education in many underserved communities — can explain the recent widening of the Black STEM representation gap.

“There’s no question that the brakes are on in terms of race-conscious affirmative action programmes,” he said. “Science and math, they were kind of the first casualties.”

Undark’s analysis suggests that the widening representation gap among Black American students is, in fact, unique to the science and engineering fields. Black students’ representation in STEM majors has declined even as their representation in non-STEM fields has continued to climb. Within STEM, the trend is pervasive across disciplines: Life sciences, physical sciences, mathematics and computer sciences, and engineering all track a similar rise and fall, with numbers in engineering and the physical sciences beginning to fade around 2000, slightly earlier than the other disciplines. Even in the life sciences, where the share of Black graduates ticked up in recent years, that proportion remained lower than it was in 2004. Black students’ share of doctoral degrees in the physical sciences, math, and engineering also levelled off in the early 2000s, after growing sharply during the 1990s, according to an analysis of data from NSF’s Survey of Earned Doctorates.

The STEM disciplines may have stood the most to lose in the retreat from affirmative action, in part, because they had benefitted richly from the programmes to begin with. The 1999 College Board report, for example, noted that “the great majority” of minority retention initiatives were focused on the sciences, math, and engineering. Only six of the 20 programmes they surveyed were “designed to support students in majors outside of science, math, engineering, and technology,” the report found — a ratio its authors said they felt confident was “reasonably consistent” with the nationwide ratio.

But even among experts who think that Black students’ declining STEM representation can be linked to the demise of affirmative action, most believe other forces are also at play. They point not only to factors like the lack of access to quality K-12 education but to rising tuition costs, declining enrollment at HBCUs, and the practice, common in science and engineering departments, of using difficult introductory courses to weed out poor performers early in their academic careers.

The NSF’s James, who declined to comment in broad terms on how affirmative action policies might be impacting Black representation in STEM, offered a litany of alternative explanations for why the numbers might be going down: the growing length of time required to complete STEM degrees; a lack of awareness, in some communities, of the available major options; a dearth of social supports in many schools; the decreasing affordability of college.

But among those factors, experts struggle to pinpoint a discrete, identifiable shift that coincided with the early 2000s turn toward falling Black representation in STEM. (James said she thinks the recent declines have less to do with programmatic shifts than with inconsistencies in universities’ approaches to achieving diversity.) When it comes to policies surrounding affirmative action, however, the correlations are difficult to miss.

Almost immediately after affirmative action bans were instituted in California and Texas, observers noticed declining minority enrollment in those state’s flagship schools and professional programmes. “Universities are feeling the impact of recently approved anti-affirmative action initiatives that ban the consideration of a student’s race in admissions decisions,” a 1998 report in The Scientist magazine noted. “Medical schools have seen a dramatic decrease in minority enrollment, [and] grad schools also have seen a noticeable decline.”

David Mickey-Pabello, a postdoctoral fellow studying ethnoracial relations at Harvard University, is part of a small community of sociologists and economists who have been working to probe the impacts of changes in affirmative action policy with statistical rigour. He uses what’s known as a differences-in-differences approach — a form of statistical regression that attempts to mimic a randomized controlled experiment — to tease out the effects of affirmative action bans from natural variation and other unrelated factors that can influence enrollment and graduation numbers.

As part of his PhD thesis, Mickey-Pabello used the differences-in-differences approach to analyze 25 years of bachelor’s degree data from more than three dozen states — some with affirmative action bans, some without. He concluded that state-wide affirmative action bans were responsible for a 12% decline in the share of STEM degrees awarded to minority students, a statistically significant drop that was larger than declines in non-STEM disciplines. Moreover, the impact of the bans appeared to grow stronger over time.

Mickey-Pabello’s study is currently being peer reviewed for journal publication, but it aligns with previous differences-in-differences studies that have identified negative effects of affirmative action bans on minority enrollments in STEM graduate programmes, medical schools, selective colleges, and public flagship universities. (One recent differences-in-differences study concluded, contrary to Mickey-Pabello’s findings, that the negative impact of affirmative action bans on minority graduation rates in STEM was statistically significant only at highly selective universities.)

Mickey-Pabello sees affirmative action bans as a form of “laissez faire racism” — a policy that advocates for colourblind meritocracy, but perpetuates racism through willful ignorance. Still, he isn’t sure that the bans alone can explain the nationwide decline in Black STEM representation. “There’s probably some other force going on there,” he says. “Would I say that affirmative action bans are part of that story? Certainly.”

Not everyone who studies these trends is convinced that the cause-and-effect is being correctly interpreted. Richard Sander, a UCLA law professor who has spoken out against affirmative action, offers a counternarrative for the widening representation gap in STEM.

Sander is a forefather of mismatch theory, which argues that race-conscious admissions policies place underprepared students at elite schools where they are likely to fail — as opposed to less competitive schools where they might succeed — and therefore have a chilling effect on minority students’ graduation rates.

That theory is widely disputed, and on its face, it would seem to predict that Black STEM representation should improve, not wane, as affirmative action programmes are curtailed. But Sander posits that the use of racial preferences in admissions decisions has become more prevalent since the implementation of affirmative action bans, despite institutions’ public declarations to the contrary. “There’s no question that at law schools, preferences have become larger and more pervasive,” Sanders said, suggesting that the same may be true of other disciplines. “Even where the bans exist — even at my university — although it’s illegal, preferences are taken for granted.”

“That’s bullshit,” says Freeman Hrabowski, president of the University of Maryland, Baltimore County. “There’s enough research that’s been thrown around, and there are enough scientists who have shown [mismatch theory] not to be true.”

Still, Hrabowski also says he doubts that the retreat from affirmative action is to blame for the stagnation of Black representation in the STEM fields. “The programmes that were at one point, just for minorities, never stopped being for minorities,” he said. “They just added on other categories — first-generation college, for example, low income. Any campus that had a commitment to African Americans could use language … to bring in more African Americans.”

But even at Hrabowski’s home institution, which is among the nation’s top producers of Black STEM graduates who go on to earn doctorates, the data suggest a more complicated story. In 1996, in the wake of the ruling against the Banneker scholarship programme, UMBC opened its vaunted Meyerhoff Scholars Program, previously limited to Black students, to participants of all races.

Despite a strategic effort to maintain the numbers of Black scholars — an effort that included securing separate funding to cover any influx of White and Asian participants — Black enrollment in the programme slipped, even as overall enrollment in the programme grew.

During the 10 years following the change, the programme averaged 20% fewer Black enrollees than it did during the five years before the change, according to a 2007 retrospective coauthored by Hrabowski. Had class sizes stayed the same “and the programme stayed race-exclusive,” Hrabowski and his colleagues wrote, “there would have been an additional 80 African American students,” roughly equivalent to twice the number of students in a typical class of Meyerhoff Scholars before the change.

The Meyerhoff Scholars Program’s final all-Black class, admitted in 1995, would likely have graduated from the university around 2000, roughly the same time that Black representation among engineering and physical science majors peaked nationally. Today, Hrabowski maintains that the drop off was due primarily to difficulties securing funding — difficulties that still persist.

When the Supreme Court affirmed the constitutionality of race-conscious policies in higher-education in its 2003 Grutter v. Bollinger decision, Shirley Malcom breathed a sigh of relief. No longer, it seemed, would she and her colleagues at AAAS have to spend their time gaming out alternatives to the affirmative action policies that had proven effective in the past.

But it soon became clear that the court had left considerable room for interpretation regarding just which circumstances permit the use of race, and precisely how race may be used. In the face of that ambiguity, and under pressure from conservative groups, says Malcom, the response of many institutions “was to pull back … pull back, as in don’t do anything.” Wrote Malcom and her coauthors in a 2004 report, “Universities are changing participation requirements beyond what might be needed to satisfy the letter and spirit of the Supreme Court rulings.”

Seven years later, Malcom, several of her AAAS colleagues, and a team of legal experts published what she calls “a different kind of document” — a 200-page handbook on diversity and the law, designed to help university counsels navigate the new legal landscape around diversity in higher education. “What we were trying to say was … the law doesn’t limit you as much as [affirmative action opponents] have said,” she said.

Malcom thinks that retreat from affirmative action is at least partially to blame for the recent widening of the representation gap. But she also thinks there is another, less-tangible factor at play. “The students are responding to the environment,” Malcom said. “They are responding to the messaging that they are not receiving. But they are also responding to the larger societal message.”

Today, the use of race-conscious policies in higher education remains a point of contention. A discrimination lawsuit against Harvard University, filed on behalf of Asian-American applicants, is currently wending its way to the Supreme Court. A similar case remains alive against the University of North Carolina.

In light of the ever-shifting landscape, Malcom and her colleagues plan to publish an update to their diversity and the law handbook. Asked if that means she thinks there remains legal leeway for institutions to do more than they’re currently doing to boost diversity in STEM, she was measured, but clear, in her response.

“In all likelihood, yes.”

Ashley Smart is the associate director of the Knight Science Journalism Program at MIT, and a senior editor at Undark.