Streetwise Professor

During his testimony before the House Ag Committee on 9 July, 2008, Rep. Bart Stupak (D-MI) asserted that current speculation in the oil markets is just like the Hunts in silver in 1980. The next day, a Republican representative (Bob Goodlatte, R-VA of Virginia, if memory serves) asked me whether the analogy was correct. (Unfortunately, hearing transcripts are not yet available.)

I responded no, they were not, and that indeed the contrast between the Hunt episode and the current oil market provides an excellent illustration of (a) why there is no evidence that speculation is distortion the energy markets, and (b) what kind of evidence you would see when such distortion occurs.

In stark contrast to oil speculators today, the Hunts accumulated huge–and I mean huge–quantities of physical silver. In September, 1979 the Hunts owned about 40mm ounces of silver in deliverable stocks, and COMEX and CBT deliverable stocks were approximately 105mm ounces. By March the Hunts owned approximately 100mm ounces of deliverable stocks, and total COMEX stocks had risen to about 130mm ounces. In addition, I have read that the Hunts also owned about 90mm ounces in Switzerland. In a nutshell, during their silver escapades, as prices were rising dramatically, the Hunts held large and rising amounts of physical silver, and deliverable silver stocks rose substantially (approximately 20 percent).

Moreover, there were manifest signs of distortions in the physical market. Massive amounts of silver coins were melted into bullion. So many coins were melted that the capacity to convert them was strained. As a result, the differential between the price of bullion and the price of coins swelled to unprecedented levels. (This differential represents the shadow price, and the marginal cost, of the conversion. The marginal cost increases with the amount of coins converted.) Large amounts of silver flowed out of India–and into the hands of the Hunts.

The prices of silver were clearly distorted in 1979-1980. These price distortions were reflected in (a) the flow of large stocks of silver into the Hunts’ hands, and (b) distortions in the physical market. This illustrates that if speculators drive up prices–if they are willing to pay more than anybody else for the commodity–they must end up holding the commodity, and that distortions in prices inevitably lead to distortions in the physical market.

None of these signs are present in today’s energy market, or other commodity markets, for that matter. Prices are indeed high, but speculators are not sitting atop large and growing stocks of the physical commodity. If they were, one could say, with Rep. Stupak, that we are seeing the second coming of the Hunts. Since they aren’t, we can’t. Indeed, as I testified in Congress, the differences between silver 1979-1980, and energy in 2007-2008 are very revealing. And what they reveal is that today one should look for other causes of high energy/commodity prices than excessive speculation.

It looks like the energy markets have dodged a bullet; the Stop Excessive Energy Speculation Act (S. 3268) has failed a cloture vote, and is in limbo despite an earlier 94-0 vote in favor of cloture on the motion to proceed. This sparked a Harry Reid hissy fit (always music to my ears.)

And a good thing too. Although not including some of the most damaging proposals that have been mooted, such as draconian increases in margin or the outright banning of large financial institutions from the energy derivatives markets, it was plenty bad enough.

Some of the bill’s greatest misses include:

• Designating some market participants as “nonlegitimate hedge traders” and limiting the designation of “legitimate hedge traders” to those entities that actually produce or consume physical petroleum or natural gas.
• Establishing an advisory committee consisting of legitimate hedge traders only to make recommendations to the CFTC on position limits applicable to nonlegitimate hedgers and speculators.
• Extending position limits to OTC markets.
• Giving the CFTC the authority to intervene in the OTC market in the event of “a major market disturbance.”

All of these measures are pernicious. The would do nothing to reduce manipulation or improve the efficiency of the energy markets or reduce prices, but would reduce the effectiveness of these markets as risk transfer and price discovery mechanisms and raise compliance costs.

The creation of a separate caste of hedgers–“nonlegitimate”–makes no sense economically. Financial institutions that use the energy markets as part of a portfolio management strategy are as legitimate hedgers as an oil company or an airline. Even the very language “nonlegitimate” is tendentious and offensive.

Moreover, it is a dangerous precedent to give one group of market users a say over what other market users can do. The favored group is likely to use its power to benefit its own members to the detriment of the unfavored ones. Note that many firms that could fit in the “legitimate hedger” category speculate too, and could exploit the powers vested in the advisory committee to favor themselves at the expense of other competitors in the supply of risk bearing services. Similarly, long hedgers (airlines, as an example) would benefit if long speculation, or long portfolio hedging, is constrained. (Long speculators offset the effect of short hedging pressure, and thereby cause the futures price to rise relative to the expected spot price. Long hedgers benefit from a fall in the futures price relative to the expected spot price, and hence would benefit from constrianing other longs in the market.) These entities could also use their position on the advisory committee to advance their interests at the expense of short hedgers and long speculators.

Since speculative limits designed to make speculation more costly are detrimental to the efficient operation of the market (by constraining the ability of the markets to shift risk), extension of these limits to the OTC market will impair the effectiveness of the markets as risk management tools. Moreover, the CFTC (and its fox-in-the-henhouse “legitimate hedger” advisory board) are charged with the responsibility of setting position limits to eliminate “excessive” speculation. Good luck with that. These limits will just become another political and regulatory football over which competing interests will squabble.

The extension of emergency authority to the OTC market is a particularly scary expansion of regulatory authority. Here’s the language in full:

GENERAL.—In the case of a major market disturbance, as determined by the Commission, the Commission may require any trader subject to the reporting requirements described in paragraph (3) to take such action as the Commission considers to be necessary to maintain or restore orderly trading in any contract listed for trading on a registered entity, including—
”(i) the liquidation of any over-the counter transaction; and
”(ii) the fixing of any limit that may apply to a market position involving any over-the-counter transaction acquired in good faith before the date of the determination of the Commission.

The definition of “major market disturbance” is extremely broad and vague:

• `(C) MAJOR MARKET DISTURBANCE- The term `major market disturbance’ means any disturbance in a commodity market that disrupts the liquidity and price discovery function of that market from accurately reflecting the forces of supply and demand for a commodity, including–
• `(i) a threatened or actual market manipulation or corner;
• `(ii) excessive speculation;
• `(iii) nonlegitimate hedge trading; and
• `(iv) any action of the United States or a foreign government that affects a commodity.

In other words, not only does this provision permit the CFTC to intervene in the event of a corner–a real manipulation–it gives the Commission the authority to interfere directly in the market whenever in its infinite wisdom it deems that “excessive speculation” (where excessive is evidently in the eye of the beholder, and is not defined and is not capable of being defined) or “nonlegitimate hedging” is distorting prices. This in effect gives the Commission the authority to second guess the market. Moreover, since regulators are subject to pressure from organized interests who may benefit from an intervention, this opens the door to all sorts of influence activities and will undermine the certainty of contract in the energy markets. Think of it; somebody makes a bad trade, begins to howl about excessive speculation distorting the markets, and just might succeed in buffaloing the CFTC into forcing liquidation of the offending contracts. Great. Read my 1995 JLE piece on self-regulation of commodity markets for a few examples of how these powers can be used to make mischief.

In all honesty I doubt that the CFTC would use these powers. But they are there, and an activist Commission or one subject to intense lobbying or Congressional pressure could well use them, and wreak tremendous havoc in the markets. This is a really bad idea.

I would oppose this provision even if its application was limited to “(i) a threatened or actual market manipulation or corner.” As I have argued for well over a decade in my academic research, it is best to address corners and real manipulations ex post. The likelihood of a wrongful intervention is much greater (due to the lack of good information and the difficult of making decisions under time pressure) when a regulator attempts to intervene while a corner is in progress. Moreover, the regulator is likely to intervene only when the price distortions resulting from the exercise of market power are large, so early intervention is unlikely to mitigate price distortions all that much. Manipulation is perfectly suited for deterrence through harm-based sanctions, like those imposed on BP in the aftermath of its propane adventure. It is inefficient to rely on emergency authority, and its extension to the OTC markets is a bad idea.

Advocates of this type of regulation make ominous allusions to the dangers of “dark markets” not subject to constant government observation and meddling. Like small children, paranoics, and control freaks, they appear to be phobic about things that consenting adults might do out of their sight.

I, for one, am not scared of the dark, or most of the things that go on in the “dark” OTC markets. One thing is for sure; if a real manipulation takes place, it will come to the light in short order. As Frank Easterbrook has written, an undetected manipulation is an unsuccessful manipulation. If somebody corners the market, it will not remain a secret, its perpetrators will be known to other market participants–and to the authorities shortly thereafter. That is the time to bring the legal guns to bear. Until that time, let adults do in the dark what they will.

The BP propane episode is a perfect example of this. It is hard to conceive of a darker, more obscure corner of the energy market than propane. It is traded almost exclusively OTC, and trading activity in that commodity is dwarfed by that for other energy commodities. BP’s actions soon sparked the outcry from affected market participants, and soon parallel civil and criminal investigations were commenced. One trader plead guilty to a felony, and sometime thereafter the company entered into a deferred prosecution agreement with the government and agreed to pay a large civil fine and criminal penalty. We’re talking 9 figures.

It is far better to devote scarce enforcement resources to investigate instances of real potential harm that occur relatively infrequently, rather than spend resources to monitor all market activity on an ongoing basis, since most of that activity is benign at worst, and beneficial at best, and hence most of the resources devoted to oversight are wasted.

In sum, Harry Reid’s pet energy speculation bill is a disastrous piece of legislation. It is built on a foundation of anti-speculation animus that has no basis in economic theory or empirical evidence. It mandates a tremendous expansion of regulatory authority that cannot be used for good, but which has the potential to be used to distort markets, to intervene when intervention is unjustified, and compromise the surety of contract. Hopefully the bill is dead. But just to make sure, let’s drive a stake through its heart and shoot it with a silver bullet and repeatedly drive an M-1 Abrams over it repeatedly.

I gave testimony on the role of speculation in energy markets before the House Agriculture Committee on Friday, 10 July. Here’s my written testimony. I will post a link to the transcript as soon as it appears.

It was an interesting experience. I doubt I changed any Congressional minds, but I think I did provide a framework that may help policymakers sort through the issues. The key lesson I tried to hammer home was that it is imperative to distinguish between the derivative markets and the physical markets. These markets perform different functions. Derivatives markets are markets for risk, and speculative and hedging activities in these markets will affect the price of risk. Absent any use of the derivatives markets to distort the physical market–as during a squeeze, for instance–even very large scale speculation in the derivatives market is unlikely to have an impact on the prices in the physical market (except insofar as that speculation is informed, and therefore influences expectations about future spot prices.)

The other point I tried to emphasize–and this is what I think made the biggest impression, at least on some–is that if “excessive” speculation truly screws up prices, these price distortions necessarily manifest themselves in the form of distortions in quantities; in inventories, in output, in consumption. The reason prices are important is that they direct the flow of resources. If prices are distorted, resource flows will be distorted too. Heretofore, there is no credible evidence of distortions in quantities. Hence, in my view, there is no credible evidence that speculation has been “excessive” in any meaningful (economic) sense of the word.

My OpEd on oil speculation ran in Friday’s (7/11) Wall Street Journal.

Index funds and pension funds are the most recent bêtes noir in the commodity speculation debate. These participants are primarily long, are supposedly relatively insensitive to price (which is an assertion, rather than something for which reliable evidence has been produced), and hence according to critics are driving up prices of oil and grain. And by “prices,” critics mean the prices of physical commodities that consumers buy.

These arguments fail to grasp important aspects of the operation of derivative and commodity markets, and ignore the benefits these market participants can reap from trading in the commodity markets.

The main error in the argument is that it does not recognize that there are multiple markets for commodities, some of which are markets for the physical commodity proper, and some of which are for commodity price risks. Derivatives markets serve the latter purpose, and fund investors almost exclusively restrict their activities to those markets, and hence affect only the prices for commodity price risks rather than the prices of commodities themselves.

Virtually all index investors and pension funds do not participate in any way in the physical market. If they are in futures, they roll their positions prior to delivery. If they gain commodity exposure through swaps, they are pure price takers, and even if their counterparties hedge some of the index exposure through the futures markets, these firms also roll prior to delivery and hence do not contribute any demand for physical commodities. Hence, they cannot affect—or in particular, elevate—the price of any physical commodity.

But, the critics reply, since these funds are buying, they HAVE TO make prices higher than they would otherwise be. In addressing this criticism, it is necessary to be precise and ask: what price? Clearly, all else equal, with more buying interest, the futures price will be higher than it would be without it (unless the supply of offsetting positions is perfectly elastic, as may be the case). But it is important to recognize that this buying raises the futures price relative to the expected spot price. Since index traders and index funds are likely uninformed, their buying does not systematically convey bullish information to the market, and hence does not affect the expected spot price; and if these trades have any information content, they should have an impact on prices, and this impact is beneficial.

By moving the futures price relative to the expected spot price, index/fund trading (a) does not elevate the prices consumers pay for physical commodities; and (b) affects the RISK PREMIUM embedded in futures prices, and hence the trend in the futures price for a particular contract up to the expiration date. In other words, the price that fund traders affect is the price of risk, not the spot price of oil or gas or corn.

In a Keynesian “normal backwardation” type model, in which speculators are not diversified and hence idiosyncratic risk affects futures prices, and hedgers are net short, the addition of new long speculators causes the futures price to rise relative to the expected spot price, reducing the discount in the futures price and the upward trend in the price. Their entry also displaces some other long speculators who find that the greater competition from the index/fund investors reduces the profits they earn from their trading. In this model, these other longs only participate if the futures price is below the expected spot price. Hence, as long as any of these traders remain, the futures price must remain below the expected spot price, but by a smaller amount. All that the entry of index funds does is replace one set of long speculators with another that bears the commodity price risk at lower cost, reduces the risk adjustment in futures prices, and therefore makes the net short hedgers better off. This is not Pareto improving—the displaced long speculators and some long hedgers are made worse off—but overall surplus rises because risk is borne at a lower cost. So, what’s the problem? None that I can see.

It is theoretically possible that that long index/fund participation can be so great that it completely displaces other long speculators. That is, fund trading causes the futures price to rise above the expected spot price, meaning that all other long speculators leave the market. Again, though—so what? This hurts those speculators and some long hedgers, but benefits short hedgers even more. (I should also add there is no evidence that this has in fact happened. There are still long speculators in the markets other than the index funds, so evidently they have not all been displaced.)

The theoretical possibility that participation of index/fund investors causes the sign of the risk premium to flip also points out that it is dubious to call these folks “speculators” in any event. They are really hedgers, in the very real sense that they are buying futures to reduce risk exposure. They are buying futures for diversification reasons—that is a risk reducing trade, and hence it is legitimately referred to as a hedge. They are not traditional hedgers, to be sure, but again—so what? Investors are people too. Why should they be denied the opportunity to use the commodity markets to reduce their risks? Why should they be considered inferior to other long hedgers who just happen to participate in the physical markets? Suspicion of these non-physical long hedgers seems to be an example of what Thomas Sowell once termed “the physical fallacy” in his excellent book Knowledge and Decisions.

Indeed, the presence of index/fund investors in the commodity markets serves to integrate the financial and commodity markets. This integration ensures that risks are priced consistently across the two markets. Indeed, the flow of funds from portfolio investors (who may include hedge funds too) from traditional equity and fixed income investments to commodity markets is likely a reflection of a mispricing of risks across these markets. In essence, this flow is serving to arbitrage away a differential in risk prices across previously segmented markets. Again—this is a feature, not a bug.

My work (with Martin Jermakyan) on electricity prices shows that incomplete integration can impose very large costs on hedgers. Electricity is probably an extreme case, but due to incomplete integration forward prices are substantially above expected spot prices. This is a symptom of inefficient risk pricing and risk allocation.

If all the markets are completely integrated, risk premia in commodity prices will be driven purely by the covariation between those prices and the market pricing kernel. The composition of market participants would be completely irrelevant. Barring one group from participating in the market would just open up a profit opportunity for somebody else without having the slightest effect on risk prices.

However, restricting the participation of portfolio investors may impede integration of the commodity, stock, and bond markets. In this case, risks will be priced inefficiently and risks and capital will be allocated inefficiently. Why is that considered desirable?

The main things to remember in this debate are: (a) since fund investors do not participate in the delivery market, they cannot be influencing physical market prices, except to the extent that by improving risk allocation they are making hedging more efficient and thereby beneficially affecting investment and stockholding decisions, and (b) the main effect of index/fund participation is to affect risk premia in futures prices (i.e., the relation between futures and expected spot prices), and in a way that improves the efficiency of risk allocation. Index/fund participation facilitates the integration of commodity and financial markets. This helps rationalize the pricing of risk. Again—a good thing.

The entire debate over speculation in commodity markets, at least the political side of that debate, is largely devoid of good economic reasoning. This goes in spades for the debate over the role of index funds and pension funds in the market. The “they buy therefore they drive up prices therefore they should be banned” syllogism is completely bogus. It is bogus because fund buying does not in the first instance drive up physical prices, and hence does not affect the price that we all pay at the pump. It is bogus because fund participation affects risk premia—the relation between futures prices and expected spot prices rather than spot prices or expected spot prices. It is bogus because it affects risk premia in a good way. It ensures consistent pricing of risk across markets and the efficient allocation of risks across markets.

Some people may be hurt by this flow of capital—but it is not the people that politicians are allegedly defending, such as individual consumers. Traditional speculators who earned rents in less-integrated markets are hurt. Long hedgers in markets dominated by short hedgers may be hurt, but their losses are more than offset by the gains of short hedgers and the fund investors themselves.

More generally, I have yet to hear anyone—anyone—present a credible case that there is a true (rather than pecuniary) externality that results from the entry of a new class of participants into the commodity markets. Absent such an externality, the free flow of capital across markets will have a beneficial effect, rather than a baleful one. Index and pension fund investors participate in the commodity markets because it improves their risk-return tradeoffs. This is a bad thing? Competition and the free flow of capital facilitates the efficient pricing of risk. This is a bad thing? These flows do not affect, let alone distort, the physical markets except insofar as improving the allocation of risk improves the efficiency of investment and inventory decisions. So this is a bad thing?

When one understands the very basic finance of index/fund participation in commodity derivatives, it is evident that restrictions on their participation in the market, either through position limits or (in the extreme) banning them altogether are extremely misguided. Extremely. Their participation should be encouraged, not limited. Derivatives markets are first and foremost markets for risk, and index/fund investors in commodities are just using those markets for the very purpose of managing risk. By doing so, they are improving the allocation of risk and its pricing across all markets—stocks, bonds, real estate, and commodities. They bear commodity price risk at a lower cost than others. This is unambiguously a good. Unambiguously. Will Congress understand this? Sadly, there is considerable room for doubt.

Over the weekend saw an interesting presentation at the Missouri Historical Society by Gari Carter. She spoke about her book, Troubled State, which is based on the diaries of Franklin A. Dick, the Provost Marshall General of Missouri during the Civil War. Leafing through the book, one passage caught my eye:

But the People of the North have become accustomed to the war–here in Phil’a the People go on just as ordinary times–They gay continue their giddy amusements–the errands go on with business as usual, & the war is an interesting topic, which they keep more less in mind.

But the rise and fall in prices, seems to be a nearer & more vital subject to most of the people–the mania for speculation in oil stocks is very great–All classes & descriptions of people here are in it, and large fortunes are lost & made rapidly in those gambling operations.

Do Bart Stupak and Joe Leiberman know about this?

James Hamilton is a world-class econometrician who has done pathbreaking work on the relations between oil prices and the real economy. Recently, he has blogged extensively about the potential contribution of speculation to the price of oil. Although he argues that most of the runup is fundamentals-driven, he is open to the suggestion that the increase in price since January is not primarily the result of fundamentals, but is instead speculative in origin. He is sensitive to the objection that a speculative distortion in prices should also manifest itself in increases in stocks. He attempts to counter this criticism with an example:

A much more important way in which the spot price of crude would affect the refiner’s demand for the product is through an intertemporal calculation. Given my customers’ demand, I’m going to need to buy the product sooner or later. If you charge me a lower price today than you’re going to charge me next month, I’d choose to buy more today to put it into inventory. If you charge me a higher price today, I’d rather run down my inventory and buy the oil next month, and of course the futures market allows me an opportunity to lock in a price for doing just that. Thus by far the most important factor in refiner’s demand for July oil will be the August futures price. If my production plans left me willing to buy July oil for $124.25/barrel when August oil was selling for $124/barrel, I’ll probably want to buy July oil for $126.25/barrel now that I’m forced to pay $126/barrel for August oil. Thus to a first approximation, the spot price would move by exactly the same amount as the near-term futures price. A $1 increase in the August futures price would shift the demand curve for July spot oil up by $1. In this fashion, an ever-increasing volume of speculative purchases of the near-term futures contracts would drive the spot price up with them.

There are two fundamental problems with this example, one logical the other empirical.

With regards to the logical problem. The premise of the hypothetical reminds me of the old Steve Martin routine “How to make a million dollars without paying taxes! First, make a million dollars.” That is, James pulls the $1 increase in the August price out of a hat, like a magician producing a rabbit. But why would an increase in speculative futures purchases lead to a dollar increase in the price of the August contract? Setting aside that the speculators might be informed–in which case, the price response is likely rational and efficient–why would other market participants believe that an (uninformed) speculator’s purchase should lead to an increase in the price of oil in August? Perhaps they anticipate that the August contract will be offset, but the speculator will buy even more for September delivery. This just kicks the can down the road a bit. Due to the ultimate connection between the futures price and the price of physical oil (that results from the delivery mechanism) the only way an uninformed speculative purchase will lead to a price increase is if market participants anticipate that some portion of the speculator’s purchase will not be offset. That instead, the speculator will take delivery of oil, and USE THAT OIL INEFFICIENTLY.

Inefficient use can occur as a result of a corner or squeeze. The speculators could squeeze some future contract, causing distortions in the production and transportation of oil. (A squeezer demands excessive deliveries, and essentially forces the delivery market up the supply curve to outputs beyond the competitive equilibrium point.) The problem here is that many speculators are unlikely to execute a squeeze, although A Speculator might do so. Multiple speculators may collectively have the ability to squeeze the contract, but they also face a collective action problem. Each one reasons: if everybody else is going to take to many deliveries to drive up prices, why should I go along? Why don’t I just liquidate my entire position at the artificially high price, and let everybody else bury the corpse (i.e., dispose of the excessive deliveries at a depressed post-squeeze price.) Individual rationality of the speculators results in a non-cooperative equilibrium where no squeeze takes place. Thus, given that myriad hedge funds, pension funds, investment banks, etc., make up the speculative crowd, unless one or two of them possess market power it is highly unlikely that anticipation of a squeeze could cause the price to rise in response to an increase in speculative purchases.

Another story is that somewhere along the line, speculators will take excessive deliveries, and just hoard the oil, keeping it off the market to enhance the price. This scenario also faces the prisoners’ dilemma issue, and another problem as well. Even for an individual speculator, it is difficult–and arguably impossible–to make a credible commitment to carry out this inefficient action. This is analogous to the Coase durable goods monopoly problem. (My 1993 Journal of Business paper discusses this issue in detail in the section titled “Pure Monopoly Manipulation.”)

That is, once he takes delivery and liquidates his futures position (at a high price), why should he just hold onto the commodity he received via delivery, reducing consumption below its efficient level and elevating price going forward? He would rationally sell the good, driving down its price to the efficient competitive level. But, forward looking counterparties, recognizing the speculator’s inability to precommit to the inefficient policy, would conjecture that he will not carry through on his threat, and hence are willing to sell the future at the competitive price today. The long would like to precommit to holding supplies off the market, but this is problematic, not to say impossible.

In a nutshell: distortion in the futures price today via the mechanism of speculative purchases requires an anticipation that the speculators will somehow take actions (a corner or hoarding) that reduce consumption below its efficient level in the future. There is no credible reason to believe that a large number of speculators can do this. Index funds that roll all their positions deterministically definitely cannot do this–they are effectively precommited NOT to take delivery. Ditto for those rading cash settled instruments, such as ICE WTI futures, or swaps (although the holders of such instruments could have an incentive to distort the physical market to enhance the value of their derivatives position.)

James’s example also poses other difficulties. The expectation that speculators will somehow take actions in the physical market that reduce supplies in the future (and such an expectation is a necessary condition for the price to move today, per the hypothetical) will induce immediate real responses. An anticipated increase in the demand for oil in the future will induce, at the margin, a decline in current consumption and an increase in current output, and an increase in stocks, in order to meet the anticipated future demand. That is, it is unrealistic to argue, as James’s example suggests, that an increase in future demand for the physical product will not lead to an increase in current inventories.

The empirical difficulty relates to the prediction inherent in the example: that there is a one-for-one relationship between the change in the spot and futures prices, especially when the market is in backwardation (as James deliberately specifies in the example, as he explains in his responses to comments). I have produced empirical evidence for industrial metals (J. of Bus., 1994) and oil (J. Futures Markets, 1996) that shows that when the market is in backwardation: (a) the correlation between spot and futures is well below one, and (b) the spot volatility is higher than the futures volatility. This implies that if one regresses the futures price change against the spot price change, the coefficient on the latter will be less than one–and often substantially so (e.g., on the order of .75 or .8). (Of course, if the regression is run the other way, the coefficient could be bigger or less than one, but in practice the coefficient is still less than one because the correlation effect dominates.)

Futures and spot move one-for-one when the market is in full carry (or nearly so)–but only because inventories (which are large when the market is in a carry) provides an intertemporal connection between spot and futures prices. This link is attenuated when the market is in backwardation. (I’ve also demonstrated this theoretically/numerically in solving dynamic storage models.) But when the market is in backwardation, the one-to-one relationship between spot and futures moves is unlikely to be observed in practice. Thus, if one treats the example seriously, as a source of empirical predictions about comovements of spot and futures prices, it is refuted by the data.

While musing about contango and backwardation, it is interesting to note that back in 2005-2006 those blaming speculation for distorting prices pointed to the strong contango as evidence that funds trading progressively more in the back end of the curve were distorting prices. These voices were notably silent when the market returned to backwardation last July. If anything, it appears that more activity is moving to the back end of the curve–so why did the contango go away? Moreover, although the price runup post-Katrina was accompanied by an increase in inventories–which is it at least superficially consistent with the price distortion story–the recent price runup was not accompanied by a jump in stocks. (And, as I said in my “Dance the Contango” post that is so loved by spam commentors, and in my recent working paper formalizing that intuition, in a world with stochastic fundamental volatility a positive co-movement in stocks and prices can occur in a rational expectations equilibrium, and hence is not necessarily a smoking gun of speculative distortion.)

In conclusion, I find James’s example unpersuasive. Although an increase in stocks that accompanies an increase in prices does not necessarily indicate speculative distortion, it is hard to imagine how the speculators could force prices above the competitive level without having to get their wingtips dirty with holdings of the gooey black stuff.

The words are, of course, Mark Twain’s. And almost a century after rumors of his death were no longer exaggerations, Congress daily works assiduously to bring his joke to life. Especially with regards to energy policy. Case in point: Rep. Maurice Hinchey (D for do you even need to ask, NY) has called for nationalization of US oil refineries. Methinks that even that ultimate cynic Twain would be amazed at such gobsmacking stupidity.

Another case in point; the mindless blather about energy speculation, and the various vacuous proposals to curb it. I’ve blogged on this subject many times, and will return to it again soon, but here I will limit myself to repeating a point I’ve made before: these assertions are based on bad economics and no evidence–always a great basis for effective policy.

From Econbrowser, I learned that Paul Krugman (oy) made a point that I was going to blog about, but my procrastination gene kicked in. Specifically, Krugman opined that if speculation were really driving up prices above the equilibrium level, we should also observe an increase in inventories. In energy, and in some other commodity markets, inventories have actually been decreasing. So Krugman concludes–reasonably–that this is inconsistent with the ceaselessly repeated mantra that speculation is causing prices to diverge widely from the equilibrium fundamentals-driven value.

There are historical examples of this phenomenon. The International Tin Council attempted to prop up the price of this metal. The ITC had to buy increasing quantities of tin, as the high price discouraged consumption and encouraged production. Eventually, inventories became so large that the ITC could no longer afford its scheme; tin was dumped on the market; and the price plunged. The accumulation of stocks of grains by governments (including the US government) as the result of price support programs are also good examples.

James Hamilton at Econbrowser is skeptical, noting that in the short run supply and demand for most commodities is so inelastic that prices can diverge substantially from the equilibrium value without leading to a detectable increase in stocks.

If forced to choose, I would go with Krugman on this. The speculation-is-driving-the-market (nuts) mantra has been heard for nigh onto three years, which is more than enough time to see an accumulation of stocks in the hands of speculators (and it would be speculators that hold the stocks as they are the ones allegedly willing to pay the inflated price). Yet this has not happened. Hamilton argues that yes, the last three years have been fundamentals-driven, but the January-February spike may not be. In such a short time frame, he notes, given the inelasticities in supply and demand it may be very difficult to detect any speculative distortion in stocks given their natural volatility. That’s a fair point, but I am reluctant to conclude, based on this absence of evidence that speculators did not cause a distortion, that prices were irrationally high in two months after being fundamentally driven for years during which speculative activity was also intense.

I do have one quibble with Krugman (about this issue–about politics, quibble doesn’t come close to covering it.) Krugman’s analysis is a static, supply-demand one. Any analysis of inventories must incorporate dynamic effects.

Moreover, a dynamic analysis implies that it is possible that stocks and prices can rise simultaneously even if all speculation is rational, i.e., in a rational expectations equilibrium. I sketched the argument in the 2006 post “Care to Dance the Contango?” There I conjectured that increases in uncertainty could cause the simultaneous increase in inventories, prices, and contango in the oil market. I’ve recently completed a stochastic dynamic programming model that formalizes the intuition from that post. (A working paper is in progress, but its completion will have to await my return from Europe in June.)

Commodity storage dynamic programming models posit that equilibrium speculative storage decisions maximize welfare. Given current inventories, the state of the economy, and the stochastic processes governing the evolution of this state, speculative storage decisions maximize the expected present value of net surplus (consumer surplus plus consumer surplus). Equilibrium prices adjust to provide maximizing individual agents the incentive to implement the optimal plan.

One proceeds by conjecturing a function that relates forward prices to the current state of the economy and carry-in. One then solves for the amount of carry out such that the current spot price equals the present value of the forward price; if the spot price with zero carry-out (i.e., complete consumption of inventory) exceeds the forward price, it is optimal to store nothing, and to have a “stock out.” The possibility of stockouts (which will occur with positive probability in equilibrium) leads to backwardations.

Conventional storage models typically restrict attention to a single, mean reverting (i.e., stationary), homoskedastic net demand shock. Other models I have worked on posit two mean reverting homoskedastic shocks, with different speeds of mean reversion.

In this most current work, I have two state variables: a mean reverting net demand shock, and a mean reverting stochastic volatility shock. That is, I don’t assume that net demand is homoskedastic; the second state variable in the economy is the volatility (or variance) of the demand shock.

This framework formalizes the intuition in “Dance the Contango.” And the results of the model are exactly as conjectured in that post. When there is a positive volatility shock, agents respond by increasing storage, and thereby reducing consumption. This leads to a simultaneous increase in inventory and price–a phenomenon that has mystified many commentators on the energy markets in recent years.

I presented the results of the analysis at a conference in Italy in January. Here are the slides, complete with some graphs illustrating the results of the model.

In sum, although a simultaneous increase in stocks and prices could be a consequence of speculative distortion–but this is not necessarily the case. I would surmise that the absence of inventory accumulation is strong evidence against the excess speculation hypothesis, but that the existence of inventory accumulation when prices are rising does not imply that speculation is distorting prices.

I understand that this is not an especially satisfactory conclusion, but it illustrates that it is very difficult to identify implications that definitively distinguish the competing hypotheses. Methinks, however, that it is necessary to go beyond the crude correlation=causation argumentation that dominates the current debate about the impact of speculation: “There is a lot of speculation. Prices are high. Therefore the speculation caused the high prices.” The exact mechanism by which this works is seldom specified. Krugman is on the right track to point out that the excess speculation hypothesis should have implications for quantities as well as prices. His Micro 101 analysis is a good place to start, but when considering storable commodities, it is necessary to model the dynamics. When one does so, one finds that his test may generate false positives. That is, in a world where the amount of fundamental uncertainty is itself random, a simultaneous increase in stocks need not be definitive proof that speculation is distorting prices. It may instead show that speculation is in fact ensuring that resources are being allocated efficiently.

It is this very difficulty of coming up with empirical tests that cleanly distinguish between the rational speculation hypothesis and its irrational speculation alternative that make it so difficult to resolve this debate. Which is why it has raged for centuries, and will continue to rage into the foreseeable future.

On Econbrowser, the University of Illinois’ Scott Irwin presents an interesting analysis of the remarkable developments in the grain/oilseed basis (the difference between the cash price and the futures price) in the past couple of years. Scott (reporting work done with colleagues Darrel Good, Phil Garcia, and Gene Kunda) shows that the basis has widened dramatically in recent years, and considers various explanations.

One he entertains, but does not strongly endorse, is that the entry of speculative interests, most notably long only funds, has driven this development. I am extremely skeptical that speculators can cause the basis to become abnormally wide during the delivery period. In particular, long only funds almost invariably roll their positions forward prior to the delivery month. (As an example, the GSCI Index rolls forward contracts that are about to expire the month prior to the delivery month.) Funds that roll forward prior to the delivery month obviously cannot be inflating prices during the delivery month.

So what is causing the periodic basis blowouts? Squeezes are one possibility. But, squeezes would be associated with (a) sharp drops in cash prices immediately following contract expiration, (b) very large deliveries, (c) a dramatic narrowing in the deferred basis (i.e., the difference between the cash price and the next-to-expire futures price), and (d) a widening of spreads between delivery locations (e.g., on the Illinois River) and non-delivery locations (e.g., central Iowa). The Irwin piece doesn’t mention whether these things have occurred.

As some of the Econbrowser commentors note, absent a squeeze, the wide basis would seem to represent a substantial profit opportunity to those operating the regular warehouses and shipping stations (the facilities where delivery actually occurs.) These firms, which include agbiz giants such as ADM and Cargill, could buy cash grain, sell futures, make delivery, and capture the basis. So why don’t they do so?

In essence, commodity trading is the business of making transformations in form, location, and time. The operators of river shipping stations regular for delivery against CBT soybean and corn contracts are in the business of transforming these products in space–from the farm to the barge on its way to the Gulf for export, primarily to Asia.

Of course firms incur costs to make this transformation. The question is: is the marginal cost of delivery so large that it equals the inflated basis? Irwin is skeptical.

Absent the exercise of market power by those who operate regular delivery facilities, however, the basis measures the marginal cost of making delivery. (By reducing throughput through their facilities to less than the competitive amount, the facilities operators would depress cash prices, elevate futures prices, and thereby cause the basis to widen to a level in excess of marginal cost. This would, however, likely require collusion among them.) So, absent evidence of manipulation or collusion, or some other exercise of market power, the basis will equal the marginal cost of the transformation from farm to barge; if the basis is above the marginal cost, competitive firms would sell futures, make delivery, and buy cash grain, thereby driving up the cash, driving down the futures, and narrowing the basis.

It should also be noted that there is somebody willing to pay the futures price and stand for delivery to obtain a shipping receipt. If the taker was not receiving something of value commensurate with the futures price, he would sell futures. (And I am very highly confident that it is not long only funds, or even hedge funds, that are standing on delivery.)

What could cause the marginal cost (and marginal value) of delivery to be so high? Here’s one conjecture that is worth some investigation. Consider corn and soybeans. These contracts are satisfied via delivery of shipping receipts. (I was on the committee that helped design these contracts in 1997.) The contract rules state that the owner of the facility issuing the shipping receipt give priority to the holders of these receipts in determining whose barges get loaded. That is, by issuing a shipping receipt, the owner of a shipping station on the Illinois River (which could be Cargill, ADM, Louis Dreyfus, Zen-Noh or one of a couple of other firms) gives up control of its facility. Put differently, by making delivery, the owner of a facility gives the taker of delivery the option to control the shipping receipt issuer’s facility. The taker has the option to choose the timing to load out the grain. The taker will behave so as to maximize the value of this option. That is, he will exercise the option to use the facility to load out grain when it is most valuable to do so–which is just when it is most costly for the operator of the facility to give up control. That is, the owner of the shipping receipt has the option to transform grain from farm to barge when that marketing opportunity is most valuable–which is just when the owner of the delivery facility would find it valuable too, but cannot exploit it because he has shorted a call on his facility by issuing a shipping receipt.

This option is valuable to the taker of delivery, and costly to the maker of delivery. Therefore, the regular firm that can issue shipping receipts will do so only if the futures price compensates it for the value of the option it provides to the taker of delivery. Asset and infrastructure owners in all commodity markets are becoming increasingly aware of the inherent optionality in physical assets used to transform commodities in space, time, and form. They are explicitly valuing these options, and making trading, investment, and resource allocation decisions based on these valuations. Thus, I would expect that operators of regular facilities take option values into account when making delivery decisions. Similarly, takers of delivery surely take these values into account.

Are option values sufficiently high to explain the wide basis? Skepticism is warranted, to be sure. It must be said that it has often been the case that “real options” theory has been used to rationalize wildly inflated valuations (see energy trading, circa 2000.) Nonetheless, it is possible that optionality has contributed to the level of, and variations in, the basis. If optionality is important, the basis should fluctuate with changes in volatility. Volatility of what? Volatility in the value and cost of making the transformation from farm to barge. Thus, higher volatility in export demand and higher volatility in domestic demand for processing should contribute to higher option values and higher basis levels. A spike in domestic processing demand (driven, e.g., by an increased demand for ethanol) reduces the derived demand for barge loading facilities; a drop in domestic processing demand increases the derived demand for these facilities. Export shocks have similar effects. Transportation cost shocks also matter. Changes in fuel prices that change the costs of shipping grain also affect the derived demand for these facilities. Variability in all these shocks, and the correlations between them, contribute to the volatility that can create option value.

Spreads–such as the basis–price bottlenecks in the transformation process. Therefore, bottlenecks are the first thing to look for when trying to explain anomalously large spreads. Sometimes these bottlenecks can be created through the opportunistic, strategic behavior of traders (as in a squeeze). Sometimes they result from unusually high demand, or disruptions in supply. The analysis presented herein suggests when there is uncertainty, spreads can price in contingent bottlenecks. Speculators will affect spreads only to the extent that they exacerbate or mitigate bottlenecks; given that most commodity funds do not participate in the delivery process, they cannot be contributing to the price of a delivery month bottleneck.

I would mention one historical case that comes to mind; it was discussed in Chapter 2 of my book (with Roger Kormendi and Phil Meguire), Grain Futures Markets: An Economic Appraisal. The cash futures delivery month basis became very wide in July, 1988 for corn, wheat, and soybeans. This was the time of a historic drought, and the operators of Chicago elevators brought in massive quantities of old crop grain in order to protect their hedges, knowing that they would not be able to attract large quantities after a shrunken 1988 harvest. Delivery warehouses were chock-full, and the Chicago cash-futures basis became very wide. In this case, the basis was pricing a bottleneck in storage space.