Big Data; Small Thinking
Even Small Data Projects Fail
20 April 2021

My experience working for two medium-sized financial institutions gave me perspectives of why firms have insecurities when it comes to “Big Data”. The definition itself was challenging as it almost defied the expression. For them, it is less about it being “big.” These intuitions have been dealing will large transaction volumes for decades. They were in essence big data companies before that term took on another meaning. Their ‘Big Data’ challenge is defined by the daunting undertaking of data integration across a sources and systems along with recruiting the talent to use it. The banks could afford the technical talent; however, the executives deemed they lacked the necessary alignment to implement a large cross-organization initiative successfully. The challenge was that these institutions had ridgdely embedded cultures that made it difficult to achieve alignment without restructuring and have exeuctives relinquishing control.

I was acutely aware of the shortcomings through my work on (small) data projects. In my eight years working for financial institutions, my responsibilities have extended to contributing to the storage and management of investment data. My experience customizing and deploying two commercial off-the-shelf portfolio management systems was woefully underwhelming. The subsequent upgrades and add-ons were even more frustrating. It was a Sisyphean task to get them greenlit. When a pound of flesh and political capital was paid to get things moving, the implementation involved a revolving assembly of IT project professionals. And when another team wanted in on the project we went from moving at a tortoise’s pace to a snail’s pace. I would have felt sorry for our vendors if we were paying them a fixed price for systems setup costs. Luckily for them, they had variable pricing.

The hierarchical structure and the ownwership in-fighting killed the potential of these small data projects. Their success required disruption, ego checks from mid-managers, and a confident analytical leader who could harmonize parites with worthy measures for success. Unfortunately, these qualities are in short supply in banking environments.