Streetwise Professor

April 2, 2014

Michael Lewis’s HFT Book: More of a Dark Market Than a Lit One

Filed under: Derivatives,Economics,Exchanges,HFT,Politics,Regulation,Uncategorized — The Professor @ 2:35 pm

Michael Lewis’s new book on HFT, Flash Boys, has been released, and has unleashed a huge controversy. Or put more accurately, it has added fuel to a controversy that has been burning for some time.

I have bought the book, but haven’t had time to read it. But I read a variety of accounts of what is in the book, so I can make a few comments based on that.

First, as many have pointed out, although this has been framed as evil computer geniuses taking money from small investors, this isn’t at all the case. If anyone benefits from the tightening of spreads, especially for small trade sizes, it is small investors. Many of them (most, in fact) trade at the bid-ask midpoint via internalization programs with their brokers or through payment-for-order-flow arrangements. (Those raise other issues for another day, but have been around for years and don’t relate directly to HFT.)

Instead, the battle is mainly part of the struggle between large institutional investors and HFT. Large traders want to conceal their trading intentions to avoid price impact. Other traders from time immemorial have attempted to determine those trading intentions, and profit by trading before and against the institutional traders.  Nowadays, some HFT traders attempt to sniff out institutional orders, and profit from that information.  Information about order flow is the lifeblood of those who make markets.

This relates to the second issue. This has been characterized as “front running.” This terminology is problematic in this context. Front running is usually used to describe a broker in an agency relationship with a customer trading in advance of the customer’s order, or disclosing the order to another trader who then trades on that information. This is a violation of the agency relationship between the client and the broker.

In contrast, HFT firms use a variety of means-pinging dark pools, accessing trading and quoting information that is more extensive and obtained more quickly than via the public data feeds-to detect the presence of institutional orders. They are not in an agency relationship with the institution, and have no legal obligation to it.

And this is nothing new. Traders on the floor were always trying to figure out when big orders were coming, and who was submitting them. Sometimes they obtained this information when they shouldn’t have, because a broker violated his obligation. But usually it was from watching what brokers were trading, knowing what brokers served what customers, looking at how anxious the broker appeared, etc.  To throw the floor of the track, big traders would use many brokers. Indeed, one argument for dual trading was that it made it harder for the floor to know the origin of an order if the executing broker dual traded, and might be active because he was trading on his own account rather than for a customer.

This relates too to the third issue: reports that the FBI is investigating for possible criminal violations. Seriously? I remember how the FBI covered itself in glory during the sting on the floors in Chicago in ’89. Not really. The press reports say that the the FBI is investigating whether HFT trades on “non-public information.”  Well, “non-public information” is not necessarily “inside information” which is illegal to trade on:  inside information typically relates to that obtained from someone with a fiduciary duty to shareholders. Indeed, ferreting out non-public information contributes to price discovery: raising the risk of prosecution for trading on information obtained through research or other means, but which is not obtained from someone with a fiduciary relationship to a company, is a dangerous slippery slope that could severely interfere with the operation of the market.

Moreover, it’s not so clear that order flow information is “non-public”.  No, not everyone has it: HFT has to expend resources to get it, but anybody could in theory do that. Anybody can make the investment necessary to ping a dark pool. Anybody can pay to get a faster data feed that allows them to get information that everyone has access to more quickly. Anybody can pay to get quicker access to the data, either through co-location, or the purchase of a private data feed. There is no theft or misappropriation involved. If firms trade on the basis of such information that can be obtained for a price that not everyone is willing to pay, and that is deemed illegal, how would trading on the basis of what’s on a Bloomberg terminal be any different?

Fourth, one reason for the development of dark pools, and the rules that dark pools establish, are to protect order flow information, or to make it less profitable to trade on that information. The heroes of Lewis’s book, the IEX team, specifically designed their system (which is now a dark pool, but which will transition to an ECN and then an exchange in the future) to protect institutional traders against opportunistic HFT. (Note: not all HFT is opportunistic, even if some is.)

That’s great. An example of how technological and institutional innovation can address an economic problem. I would emphasize again that this is not a new issue: just a new institutional response. Once upon a time institutional investors relied on block trading in the upstairs market to prevent information leakage and mitigate price impact. Now they use dark pools. And dark pools are competing to find technologies and rules and protocols that help institutional investors do the same thing.

I also find it very, very ironic that a dark pool is now the big hero in a trading morality tale. Just weeks ago, dark pools were criticized heavily in a Congressional hearing.  They are routinely demonized, especially by the exchanges. The Europeans have slapped very restrictive rules on them in an attempt to constrain the share of trading done in the dark. Which almost certainly will increase institutional trading costs: if institutions could trade more cheaply in the light, they would do so. It will also almost certainly make them more vulnerable to predatory HFT because they will be deprived of the (imperfect) protections that dark pools provide.

Fifth, and perhaps most importantly from a policy perspective, as I’ve written often, much of the problem with HFT in equities is directly the result of the fragmented market structure, which in turn is directly the result of RegNMS. For instance, latency arbitrage based on the slowness of the SIP results from the fact that there is a SIP, and there is a SIP because it is necessary to connect the multiple execution venues. The ability to use trades or quotes on one market to make inferences about institutional trades that might be directed to other markets is also a consequence of fragmentation. As I’ve discussed before, much of the proliferation of order types that Lewis (and others) argue advantage HFT is directly attributable to fragmentation, and rules relating to locked and crossed markets that are also a consequence of RegNMS-driven fragmentation.

Though HFT has spurred some controversy in futures markets, these controversies are quite different, and much less intense. This is due to the fact that many of the problematic features of HFT in equities are the direct consequence of RegNMS and the SEC’s decision (and Congress’s before that) to encourage competition between multiple execution venues.

And as I’ve also said repeatedly, these problems inhere in the nature of financial trading. You have to pick your poison. The old way of doing business, in which order flow was not socialized as in the aftermath of RegNMS, resulted in the domination of a single major execution venue (e.g., the NYSE). And for those with a limited historical memory, please know that these execution venues were owned by their members who adopted rules-rigged the game if you will-that benefited them. They profited accordingly.

Other news from today brings this point home. Goldman is about to sell its NYSE specialist unit, the former Spear, Leeds, which it bought for $6.5 billion (with a B) only 14 years ago.  It is selling it for $30 million (with an M).  That’s a 99.5 decline in market value, folks. Why was the price so high back in 2000? Because under the rules of the time, a monopoly specialist franchise on a near monopoly exchange generated substantial economic rents. Rents that came out of the pockets of investors, including small investors.  Electronic trading, and the socialization of order flow and the resultant competition between execution venues, ruthlessly destroyed those rents.

So it’s not like the markets have moved from a pre-electronic golden age into a technological dystopia where investors are the prey of computerized super-raptors. And although sorting out cause and effect is complicated, the decline in trading costs strongly suggests that the new system, for all its flaws, has been a boon for investors. Until regulators or legislators find the Goldilocks “just right” set of regulations that facilitates competition without the pernicious effects of fragmentation (and in many ways, “fragmentation” is just a synonym for “competition”), we have to choose one or the other. My view is that messy competition is usually preferable to tidy monopoly.

The catch phrase from Lewis’s book is that the markets are rigged. As I tweeted after the 60 Minutes segment on the book, by his definition of rigging, all markets have always been rigged. A group of specialized intermediaries has always exercised substantial influence over the rules and practices of the markets, and has earned rents at the expense of investors. And I daresay it would be foolish to believe this will ever change. My view is that the competition that prevails in current markets has dissipated a lot of those rents (although some of that dissipation has been inefficient, due to arms race effects).

In sum, there doesn’t appear to be a lot new in Lewis’s book. Moreover, the morality tale doesn’t capture the true complexity of the markets generally, or HFT specifically. It has certainly resulted in the release of a lot of heat, but I don’t see a lot of light. Which is kind of fitting for a book in which a dark pool is the hero.


Print Friendly, PDF & Email


  1. Two points
    Institutions are, ultimately “the small guy”. Even though they often don’t behave like that, at least in theory they have a fiducary duty to their ultimately individual investors. So it’s still skimming money from the small guy.

    That said, from what I saw so far Lewis ignores what was happening in the last few years, when (to my, admittedly limited) knowledge HFT profitability plunged quite a bit, and as you point out there are ways to deal with it. Although some practices are likely to be deemed illegal even now (at least in the UK price manipulation is illegal and there was a case recenly of a guy manipulating Brent futures ending in front of the court…)

    Comment by vlade — April 3, 2014 @ 3:23 am

  2. Great write up. Always a fan of lewis but there will always be winners and losers from change and small investors have benefitted more than institutions in thus case.

    Comment by carrytrader — April 3, 2014 @ 4:43 am

  3. I strongly recommend reading the book. Lots of interesting things in there. Most importantly, unsaid but alluded to is that it is the regulators are the guys behind this. If markets are rigged, it is the regulators that own that. If the playing field is unequal, the SROs have created that.

    The concept of fiduciary duty will be an interesting one if applied to exchanges in this instance.

    As for rigged, I think the difficult part of challenging this argument is hundreds of days without losses, perfect trading quarters. In other words enviable track records that seem to incredible to be true. The other guy with that kind of a record was Madoff. He may have been a fraud, but he was also aided and abetted by the regulators through their less than benign neglect.

    I agree that HFT is doing their job, exploiting every advantage they are provided with. I think the ugly underbelly of the story is the incompetence, lack of foresight, regulatory capture and failure to do the job amongst regulators and sro’s Lewis’ book touches this frequently without saying it.

    Comment by tw — April 3, 2014 @ 7:21 am

  4. @tw: As for the question you raise about “hundreds of days without losses,” I suggest that you take a look at this explanation by Matt Levine over at Bloomberg View:

    In addition, I found this deconstruction of Virtu’s business model, as reported in their S-1, to be very helpful. It was written by CNBC’s Bob Pisani:

    I think you will find that there is a very reasonable answer to your question.

    Comment by JAO — April 3, 2014 @ 8:26 am

  5. Well said. If this be rigging, then markets have always been rigged.

    I have a charming image in my mind of “computurized super-raptors”; it may have been influenced by the special section in the Economist this week.

    Comment by mutant_dog — April 3, 2014 @ 9:17 am

  6. It’s very unhelpful that a legitimate activity is demonised as front running. As you say, not only is what Lewis describes (AIUI) not front running, it’s not trading off MNPI, either.

    Where this sort of error leads – once the let’s-draft-a-crap-kneejerk-regulation bandwagon picks up speed – is to the situation whereby an oil company monitoring tanker movements is no longer allowed to place arb bets. The information it has painstakingly gathered gets accidentally included into the definition of MNPI.

    I haven’t read the book either, but it sounds like it is short of a useful discussion of the differences between inside, superior, and proprietary information.

    Comment by Green as Grass — April 3, 2014 @ 9:38 am

  7. vlade: Some institutions represent small investors, others represent large investors. But to the extent faster market reaction to institutional orders hurts one institution, it benefits whomever is on the other side of the trade. Only a tiny fraction of that transfer is “skimmed” by HFTs, and that total skimming is much less than was skimmed in pre-HFT days by slower market makers. The unambiguous winner is the person whose entire desired trade can be executed in a single order at the NBBO, or matched at mid. These are the small guys.

    Comment by Aaron Brown — April 3, 2014 @ 9:43 am

  8. @Green. I hear you loud and clear. And re your situation involving the oil company: the Euros are going do try to do just that, with their “inside trading” rules on commodities. It’s beyond asinine, and a compliance/enforcement nightmare.

    The ProfessorComment by The Professor — April 3, 2014 @ 11:47 am

  9. It’s not new to those of us in the business, but it is new to the public. Lewis book is shining a light on poor market structure. In interviews, he says our market structure is screwed up. I cannot speak for stock markets but futures markets are not competitive anymore. Bids and offers on the screen may or may not be there. Exchanges give rebates and special deals to some traders, without extending them to the entire market. For example the market maker program in the deep back months of the Eurodollars. No one new can come in and compete.

    Speed isn’t the issue. It’s the fact that traders are going at different speeds. Exchanges deliberately set it up.

    Just because we always have done things this way isn’t an excuse. Innovation ought to make things better. It’s made some things better, but it’s really screwed up other things. In economic terms, consumer surplus is going to fewer and fewer firms in the futures markets.

    Comment by Jeff Carter — April 3, 2014 @ 1:12 pm

  10. I have read a few complaints about Flash Boys, and as nearly as I can tell, they all seem to revolve around “it isn’t a big deal” and “it was always been this way”. Nobody seems to want to say “it’s fine, that’s the way it should work”. Is it not front running because no fiduciary responsibility was broken? Is it not insider trading because the ones with the non-public information aren’t insiders? Is it no big deal because its peak is past and it isn’t the mom and pop traders that are being hurt? I think that opinions on these may differ, but that’s just what they are, opinions and it is up to the SEC and the courts to have final say. But in the mean time, what is the social good that comes from holding a stock for 10 seconds or less? What does it accomplish other than being a drag on the people who are actually buying and selling a stock?

    Comment by Brian Utterback — April 3, 2014 @ 1:30 pm

  11. TW: “He may have been a fraud, but he was also aided and abetted by the regulators through their less than benign neglect.”

    Oh no, it’s much worse than that. One of our regulatory examiners got chatty during one of our annual exams and told me that he looked into their files on Madoff. The SRO wanted to investigate him, but a call from a friendly politician ended every attempt. In fact, that’s one of the examiners’ frustrations. When nefarious activity is suspected by an examiner, they write it up and it goes through several layers of investigation before finally going to the SEC. At each layer, the firm has a chance to respond. Politically connected firms suspected of outright fraud simply call on their connections and all investigations are immediately dropped.

    Regulation doesn’t prevent fraud, doesn’t bring it to light and doesn’t punish it. Regulation institutionalizes fraud and hides it deeply away from the public’s scrutiny….until it gets so big it can no longer be contained.

    Unlike Jeff Carter, I can speak to equity markets and to options markets. I agree that the structure is broken. I contend that the broken structure begins with a protected group called Market Makers (and the old specialist system) who get tax breaks in the form of mixed straddle for options market makers (not available to non-market makers) and the list of exemptions for both (although stock market makers have few left). Pretty much all the regulation on the books simply prevents people from engaging in perfectly normal and moral activity in their own interest. That dries up liquidity, so Market Makers are exempted in order to prevent the downside of illiquid markets. That’s rigged, my friends. Yet, somehow the brokers still roaming the shrinking NYSE floor constantly whining on CNBC have convinced the retail crowd that they were advantaged by these floor market makers “looking out” for them. Yeah, floor crowds were looking out for dumb customer orders to take full advantage.

    Most of the regulations need to be scrapped and tax structures favouring a protected interest in the form of options market makers need to be abolished. There are different rules for everyone and everyone lobbies the regulator and clowns in the Potomac Swamp for more favours. And average joe retail is just as happy to join that hue and cry for more regs and in that way unknowingly working against himself.

    Comment by Methinks — April 3, 2014 @ 2:07 pm

  12. “And this is nothing new. Traders on the floor were always trying to figure out when big orders were coming, and who was submitting them”


    Order flow is essential information for anyone making a market (whether acting in an official DMM capacity or not). In fact HFT is more disadvantaged in this respect than the old floor trading crowds.

    In the old days, you didn’t have to give up as much information to read order flow. You just had to sit and wait because the order flow came to you. Today, the HFT strategies to ferret out information forces them to give up some information. It imposes costs and exposes them much more than market makers on the floor. In many ways, HFT provides better liquidity with less downside to customers but with much greater risks to themselves. The quality of the profits are not as good.

    Comment by Methinks — April 3, 2014 @ 2:15 pm

  13. All these supposedly knowledgeable comments by self selected experts who have not bothered to first read the book they are so eager to criticize… If these are the kind of the people who defend HFT, then the case against it is rather obvious.

    Comment by Don — April 3, 2014 @ 2:25 pm

  14. Don, we watched the 60 Minutes fiasco and if there is no fast response than the disgusting false meme of “rigged” becomes truth. And reading the book now out proves we r right. But thanks for the deep wisdom.

    Comment by Cliff — April 3, 2014 @ 6:22 pm

  15. @Don-I was upfront. Consider my post a response to what has been reported about the book, which presumably reflects the most important claims therein. I have skimmed the book, and a lot of it is old news. An underground cable was laid to cut small fractions of a second from Chicago to NY. Who knew?

    I do not consider my post a defense of HFT. I consider it an effort to provide some balance and context, and to approach the subject in a less didactic and moralistic way than Lewis does. And I don’t have to read the book to know that my characterization is a fair one. I saw 60 Minutes and have read interviews he has done about the book. It is precisely his moralism and certitude about a complex and often ambiguous subject that sheds heat rather than light.

    The ProfessorComment by The Professor — April 3, 2014 @ 8:13 pm

  16. Here is what I believe (and suggested to an SEC officer)
    a) if someone wants to jump in front of a bid or offer – make them improve by a quarter of the spread. not this ridiculous .00001 cent. a quarter of the spread is material in benefiting the liquidity provider.
    b) if someone places a limit order, they need to hold it for a second. effectively enforce this by charging a fraction of a cent for placing a limit order. it will be expensive to constantly show and remove liquidity

    Comment by Kapil Khetan — April 3, 2014 @ 8:50 pm

  17. I remember more than a decade ago the head of Island ECN basically told the WSJ that he paid for order flow so he could front-run customer orders. Or to put that another way, brokers with “fiduciary duties” to their clients violated them by laundering their front-running through Island ECN, which kicked back a percentage of the front-running profits to brokerages as “payments for order flow.” It was crooked then and modern variations are not less crooked. All the so-called regulators are entirely captured by industry players and there are only two kinds of competition in the industry, rent-seeking (who can most effectively suborn regulators) and betrayal (who can cheat their clients most effectively). Stuff like dark pools and multiple exchanges is all about cheating the clients– with modern communications and computers an honest market would have a single central limit-order book and clocked batch auctions setting a single “market price” at regular (fraction of a second) intervals. Retail brokers would not be allowed to trade “for their own accounts” at all.

    Comment by Sourpuss — April 3, 2014 @ 11:20 pm

  18. actually exactly my points –

    it is a fight between the biggies, institutional investors, exchanges and HFT farms. Small investors are hardly effected in negative way – may be in a second round effect if some fund manager handling your retirement fund is impacted. But still nothing is back or white. If front running or similar predatory trading without fiduciary duty is unethical or not is debatable. But active trading like momentum ignition may not be so ambivalent, in principle not much different than rumor-mongering pump and dump.

    Comment by prodiptag — April 4, 2014 @ 12:50 am

  19. One area that hasn’t received much attention is liquidity. In the old days with high commissions, wide spreads and thin or none existent Chinese Wall between prop and trading desks brokers were willing to provide liquidity. As spreads and commissions fell and better compliance was put in place the provision of liquidity came under pressure. As brokers started operating dark pools, in which they participated for their own advantage and HFT they were able and willing to provide liquidity. While no fan of HFT and dark pools I wonder if market liquidity will suffer as they are curtailed.

    Comment by Richard Fairgrieve — April 4, 2014 @ 4:46 am

  20. The exchanges are selling access to information to people who can monetize it. This is no different than their selling price feeds to subscribers before they were accessible to the general public.

    I can talk about a number of illegal activities going on in the financial markets that are known to regulators but being ignored because of politics. It amuses me that subjects like HFT are demonized and used to create the impression the regulators are being diligent when vast sums are being gained through illegal schemes that no one wants to talk about. The regulators get to claim they are tough on Wall Street until they get hired by the Wall Street firms and get to cash in, the Wall Street firms get to bemoan how tough the regulators are while all the time they are raking in billions from various schemes and the public believes they are bring protected while they are actually being fleeced. In the end, everyone gets what they want and the wheel keeps turning.

    Comment by Charles — April 4, 2014 @ 7:27 am

  21. @Kepil,

    So, what you’ve suggested to the SEC is rules that favour market makers over customers. because:

    a.) if the razor-thin edge disappears if the MM improves the bid or offer by a quarter of the spread, then he will not better the bid or offer by any amount and the customer will have to pay more to trade (the spread he crosses will be wider).

    b.)You want anyone sending a limit order to take more risk by forcing the order to sit on the book regardless of any changes in market conditions and you think this will be costless. In reality, there will be less liquidity and more “gapping”. That is, the price moves will be much wilder. And forget about trading close to any kind of information release. Mandatory quoting periods will mean that all liquidity evaporates around predicted news like earnings releases. The only markets you’ll find will be what we call “zero bid at eff you”.

    Comment by Methinks — April 4, 2014 @ 9:32 am

  22. @Richard Fairgrave,

    You don’t have to wonder too hard. The answer is “yes”.

    Charles, I’d go further with your first example. Exchanges sell real-time datafeeds. The feed available for free is as stale as month-old bread because it’s 15 minutes old and completely useless for trading.

    Comment by Methinks — April 4, 2014 @ 9:43 am

  23. […] Lewis’s HFT Book: More of a Dark Market Than a Lit One (Streetwise Professor) • It’’s a living (Book Forum) • Audit Notes: WSJ’s Obamacare frame, undercover, too […]

    Pingback by 10 MidWeek PM Reads | The Big Picture — April 9, 2014 @ 2:30 pm

  24. You start by saying “I have bought the book, but haven’t had time to read it.” You conclude by saying “In sum, there doesn’t appear to be a lot new in Lewis’s book. Moreover, the morality tale doesn’t capture the true complexity of the markets generally, or HFT specifically. It has certainly resulted in the release of a lot of heat, but I don’t see a lot of light.” How can you come to conclusions like there is neither anything new or that the account is overly simplistic when you HAVEN’T EVEN READ IT ?

    Comment by John Banerjee — April 23, 2014 @ 9:44 am

  25. Hi,
    I think, like many things in life, diversity is the key. In that way, if one method stops working you still have lots of others to work with. I really enjoyed this post.

    Comment by Embankment Stair — August 15, 2017 @ 6:34 am

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress