Streetwise Professor

December 5, 2023

Luddism in the Oil Futures Markets

Filed under: Commodities,Derivatives,Economics,Energy,Exchanges — cpirrong @ 1:06 pm

The old, old game of Pin the Tail on the Speculator has been updated. According to Bloomberg, the speculators who now “disrupt” the oil markets are not human: they are bots. Specifically, bots operated by Commodity Trading Advisors (CTAs).

This argument consists of two parts. The first being that the crude oil futures markets have been disrupted. The second being that the CTAs are the cats behind the disruption. Both plinths are defective.

Insofar as disruption is concerned, Bloomberg claims “Trading oil has perhaps never been more of a roller coaster ride than it is today.” Further:

Just in the past two months, prices threatened to reach $100 per barrel, only to whipsaw into the $70s. On one day in October, they swung as much as 6%. And so far in 2023, futures have lurched by more than $2 a day 161 times, a massive jump from previous years.


Never more of a roller coaster ride? Well, let’s do something crazy. Like look at historical data.

The conventional measure of the wildness of the ride is volatility. The annualized daily volatility of crude oil during the alleged Rule of the Bots (the last two years) is 41.62 percent. The historical volatility (2010-2020) is 41.2 percent. (This omits 4/20/20 and 4/21/20, the day of the negative oil price and the following day.) Excluding the COVID months of 2020 produces a somewhat lower vol of 36 percent, not that much smaller than in the last two years. Further, extended excursions of realized volatility to above 40 percent are not unusual in the historical record. So to say that oil prices have been more volatile recently than has historically the case is categorically false.

With respect to the big daily moves, the Bloomberg analysis is fatally flawed because it looks at dollar price moves: big dollar price moves are more likely when prices are high than they are low, and by historical standards oil prices have been high in the last several years. It is appropriate instead to focus on percentage price changes (which is how vols are calculated, btw).

Rather than count the number of times an arbitrary threshold (like $2/bbl) is breached, it is more rigorous to look at a statistical measure of the frequency of extreme events: the “kurtosis.” Kurtosis bigger than zero means a distribution has fat tails relative to a Gaussian (“normal”) distribution, i.e., extreme moves up or down are more likely than under a normal distribution. The bigger the kurtosis, the more likely extreme moves are, i.e., the fatter the tails of the distribution.

Looking at the kurtosis of daily percentage changes rubbishes the Bloomberg analysis. The kurtosis in the last two years is 4.15, whereas from 2010-2020 it was 27.9! That is, the frequency of extreme daily price moves in years of alleged CTA disruption is far, far smaller than was the case prior to their alleged emergence as the dominant force in the. markets.

Interestingly, the kurtosis of dollar price changes is not that different between eras: 6.9 post-2020 vs. 7.2 2010-2020. So even extreme dollar price moves are less frequent in the alleged CAT era than previously. The difference is smaller, which demonstrates the need to take into account the level of prices in an analysis of “extremes.”

So the predicate for the article–that oil prices have been unusually volatile and unusually susceptible to extreme moves in the past couple of years–is not supported by the data.

As for the alleged causal factors, the dominance of CTAs is not evident in the data. CTAs are included in the “Managed Money” category of the CFTC’s Commitment of Traders Report. Here is a graph of the net position of Managed Money going back to 2006:

There was a peak in 2017-2018–a drilling boom in the US, to which I will return shortly–followed by a decline–a drilling drought–followed by a rebound to levels comparable to the 2017-2018 levels. Indeed, managed money net positions have actually been relatively low in the past year (with the exception of a recent spike) as compared to the post-2015 period as a whole. Certainly no Alice to the moon spike in CTA presence apparent here.

Bloomberg claims that the CTAs have become dominant in large part due to a sharp decline in producer hedging:

That coincided with the collapse of another source of futures and options trading: oil-production hedging. During the heyday of shale expansion about a decade ago, drillers would lock in futures prices to help fund their growth. But in the aftermath of the pandemic-induced price crash, a chastened US oil industry increasingly focused on returning cash to investors and eschewed hedging, which can often limit a company’s exposure to the upside in a rising market. By the first quarter of this year, the volume of oil that US producers were hedging by using derivatives contracts had fallen by more than two-thirds compared with before the pandemic, according to BloombergNEF data.

It should be noted that this claim that CTAs have achieved greater dominance due to an ebbing of hedging is implausible on its face. Futures are in zero net supply. If producers have reduced their net positions, necessarily non-hedgers–including CTAs–must have reduced their net positions.

Hedging has indeed declined. In the oil market, much (if not most) producer hedging is via the swaps market rather than direct producer participation in the futures market. Banks buy swaps from producers, and then hedge their exposure by selling futures. Here is a chart of net Producer and Merchant Plus Swap Dealer exposure from the CFTC COT data:

Note that there was a big increase in hedging activity (by this measure) in 2017-2018 that was reversed, followed by a partial resurgence, but in the last couple of years hedging activity has indeed ebbed, and reverted to its 2016 levels.

But note that this pattern of hedging mirrors closely Managed Money net positions. As is necessarily the case. If there is less hedging, speculators necessarily hold smaller positions. Meaning that this statement is nonsensical:

The recent wave of dealmaking by US oil producers threatens to further accelerate the decline in hedging. And it’s highly likely that CTAs will continue to fill the vacuum left by those traditional market players.

It’s not as if CTAs–or speculators generally–are “fill[ing] a vacuum.” If hedgers reduce positions, speculators do too.

The Bloomberg writers may dimly glimpse the truth, though they don’t realize it.

How did CTAs come to become so dominant? Like many current phenomena, the answer starts in the depths of the pandemic.

As shutdowns engulfed the world in 2020, fuel consumption collapsed by more than a quarter. All hell broke loose in the crude market. The benchmark US oil price briefly dropped to minus $40 a barrel and investors were in wholly new territory. Some funds that took longer-term views based on supply-and-demand fundamentals quickly pulled out.

Such bear markets proved to be “extinction events” for traditional funds, which made way “for algo supremacy,” the bulk of which are CTAs, said Daniel Ghali, senior commodity strategist at TD Securities. Russia’s invasion of Ukraine gave the CTAs another foothold. Spiking volatility in the futures market drove many remaining traditional investors to the exits, and open interest in the main oil contracts tumbled to a six-year low.

So if CTAs have indeed become more prevalent, it is because they have supplanted other speculators who exited the market. Futures are risk transfer markets. If some of those who previously took on the risk from hedgers have exited the market, either hedgers must hedge less or other speculators must step in. It seems that both things have been happening.

That’s not some ominous development–it’s markets at work. And CTAs shouldn’t be damned–they should be praised for stepping into the breach.

And another paragraph in the Bloomberg article suggests at what is actually happening here:

The unpredictability of this year’s market swings haven’t been kind to human traders, many of whom are making less money on oil than they did last year when they raked in record gains, according to market participants.

What is likely driving this story is whinging by the traditional specs, who have been outcompeted by the bots. “No fair! They are making money and I’m not! They must be cheating.”

Reminds me of my epigram from my manipulation book, where riffing on Ambrose Bierce’s Devil’s Dictionary I wrote something to the effect that a market is manipulated when it moves against me.

Again, this is markets at work. The fact that bots are doing well relatively to trad specs means that they are better at predicting market movements, or have lower costs of bearing risk, or both.

It does not mean that they are making the markets move.

So this Terminator Tackles the Oil Market narrative is really nothing more than Luddism. A new technology outcompetes the old. The incumbents complain. End of story.

Print Friendly, PDF & Email


  1. 1. Thought ‘Prof seems to be on the same wave length as George Soros’ 2nd wife, her personal motto is reportedly ‘zero tolerance for bullshit”.
    2. Luddism? a) b),employment%20and%20deteriorated%20the%20working – so far I knew only about the arts and crafts movement…thought it may come from latin ludus, bug that’s got only one d….

    Comment by Mikey — December 5, 2023 @ 3:03 pm

  2. “Kurtosis bigger than zero means a distribution has fat tails”

    I am ill, old, and tired. Yet you remind me pleasurably of a cartoon seen decades ago illustrating kurtosis with a sketch of a heavy-tailed kangaroo. Does anyone here know the source?

    Comment by dearieme — December 5, 2023 @ 3:53 pm

  3. Sorry to hear you’re not well, dearieme. You’re an entertaining commentator on several blogs and I hope you’ll continue.
    As for volatility in futures markets it’s always been there. Almost the point of it. (NB I’m a total ignoramus on the mechanisms.) So someone’s bot (algorithm) is better than someone else’s. It doesn’t mean we’re all doomed by AI.

    O/T much talk of insider trading just before Hamas invasion. But oil price didn’t move much. Thoughts?

    Comment by philip — December 5, 2023 @ 6:46 pm

  4. Everything I’ve read about fund managers (CTAs) leads me to believe they are mostly engaged in some variation of trend following strategy which means their positions are in place for days to weeks or sometimes months. That type of trading could hardly contribute to intraday volatility. In the old days everything was blamed on the floor traders, now it’s the bots.

    Comment by Andrew Stanton — December 6, 2023 @ 9:07 am

  5. @Dearieme – I hope this image will cheer you up!

    Comment by [email protected] — December 6, 2023 @ 7:23 pm

  6. @ Professor

    Your first chart would probably be better if it plotted share of OI held by CTAs over time. The absolute numbers of lots have gone up, but the OI has gone up secularly too. I suspect that on a share-of-OI basis, their relative participation has changed hardly at all.

    One frustrating thing about this cyclical anti-speculator article (because there really is only one, and it really does keep coming around) is that it is never sure whether it is speculators’ positions it objects to, or speculators’ provision of liquidity (which may produce no positions). The article usually gets around at some point to noticing that if you compare daily volume to daily OI change, most trades – shock, horror – don’t create or take off a position. All the volume is therefore wicked intraday scalping by bots; it’s not like anyone ever rolls a position from one month to another, or anything like that.

    Comment by Green as Grass — December 7, 2023 @ 4:29 am

  7. Wait until it’s only artificial intelligence machines in the market….or quantum computing programs……I do remember the switch from manual open outcry to electronic. I certainly noticed big differences in market action. First, the speed advantage I had as a floor trader was gone and I became “retail”. Second, the pit used to have a lot of resting orders that you could lean on to spread, or to try and get an edge. Gone. The claim that “bots” are disrupting the market is absolutely without merit in 2023. Oil trading went electric in what year? Bots have been the main force in that marketplace ever since.

    Another thing, most speculators do not hold positions. They grease the market daily and go home flat, or relatively flat. Long term speculators, which CTA’s would be, don’t really trade a ton once they have their position in the market. They might trade opportunistically in and out a bit, or they might press a winning position-trim a loser hoping for a change in sentiment, but the big specs I knew owned em and the only time they really could cover was when rollover happened or there was some big report and volume came into the market. If a big spec or CTA like Memphis Charlie had to get out of a position, it was going to really push the market.

    Comment by Jeff Carter (@pointsnfigures1) — December 7, 2023 @ 7:08 am

  8. Thank you, Philip, thank you, [email protected]. I am duly cheered up.

    Comment by dearieme — December 7, 2023 @ 12:46 pm

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress