"Data scientists aren't quite unicorns, but they're close": How to avoid a big data nightmare

Big data is no longer just the domain of big companies. As the perception of big data moves from futuristic hype to real-world opportunity, the promise of improved decision making, increased operational efficiency and new revenue streams has more organisations actively engaging in data analysis projects than ever before. That no longer only means more enterprise organisations, either. Midmarket companies are jumping on the big data bandwagon in a major way.

In fact, a recent survey by Competitive Edge Research Reports indicates that an astounding 96 per cent of midmarket organisations are either already in flight with a big data initiative, or plan to start one in the next year. That's a whole lot of companies whose big data projects are either going to sink or swim in the very near future.

One of the benefits to being a company that arrives late to the game in the adoption curve of any technology cycle is the opportunity to learn from those who came before you, and in the case of midmarket companies about to embark on a big data project, their enterprise forerunners have left a trail of lessons learned. By learning from the mistakes of the enterprise and taking steps to avoid them, midmarket organisations can position themselves to enjoy the immediate success that may have eluded their larger competitors. Here are the four most important of those lessons.

Lack of alignment with executive stakeholders will derail any project

Data analysis done right is not about technology; it's about business. Before you start any big data analytics project, you first need to secure the support of the company's executive stakeholders. There are two primary reasons for this. First, these stakeholders are the ones who can ensure that you have the resources you need – whether it's the right team, the required budget, or the necessary data access – to position your project to succeed. But there's an even more important reason. Data analysis is only effective if someone is willing to act on it. This is a lesson many enterprises learned the hard way. If your key executives aren't prepared to make tangible business decisions based on the findings of a big data project, the project itself will have served no purpose.

Don't fixate on infrastructure savings

Many big companies initially thought that moving their archive data off legacy databases with expensive license requirements and onto the nearly free clusters of databases, such as Hadoop, would yield significant cost savings. While shifting data to these unstructured sources can in fact save your company on licensing costs, the labour required to architect, deploy and manage these systems can be significant – so significant that many large companies are finding that all they've done is shift costs from licensing to labour.

The takeaway for midmarket companies is this: Factor labour costs into your ROI calculations, but more importantly, don't fixate on infrastructure savings to begin with. Focus instead on outlining and answering questions that are critical to your business. That's where true cost savings are ultimately found.

Data scientists aren't quite unicorns, but they're close

Simply put, labour requirements in the big data realm are difficult to satisfy. Though new educational programmes are now being created with increased regularity, universities and professional training services were not initially equipped to handle the tremendous demand for so-called data scientists. The number of people needed to support the deployment of big data technologies has overwhelmed the pool of IT resources. If deep-pocketed enterprise companies can't go out and hire the talent they need, chances are you won't be able to either.

There simply aren't enough data scientists in the world today, nor will there be in the foreseeable future. Instead of focusing on finding a single data scientist, you should instead focus on building data science teams from within your organisation. Train team members in-house to manage your customised big data initiatives. Find enthusiastic DBAs and business analysts willing to learn and take the next step, and offer the on-the-job training they need to take it.

Collaboration is key

This is another lesson most enterprise organisations have to learn the hard way. Line of business leaders in marketing, sales and other functional departments were led to believe that they could successfully embark on data analysis projects without the help – and in some cases, without the knowledge – of IT. As they soon found out, however, while they might be tremendous innovators, line of business leaders are not equipped to manage, govern and scale data analytics products. Roadblocks were inevitably hit.

With the benefit of hindsight, midmarket companies can avoid this pitfall by making collaboration between IT and lines of business a priority. Not only will this ensure that data is properly governed, and systems are properly managed and can be scaled when needed, but also that the right people have access to the right data at the right time. A data analytics project is of little value if you can't trust the validity of its findings. That trust is far easier to come by when collaboration is taking place.

Clearly, we've reached the point of no return with respect to big data and analytics. Organisations that wish to remain viable and avoid being outmoded by their competitors or outgrown by their customers need to embrace a data-driven approach to management and decision making. There was a time not long ago that this would have seemed like a road fraught with pitfalls. Today, however, the vast majority of those pitfalls have been exposed, and the path to success is clear. For midmarket companies, it's now just a matter of following it.

Joanna Schloss is a business intelligence and analytics evangelist at Dell Software


The top antivirus programs for your iPhone

With the launch of new age smartphones, security risks have literally increased tenfold. Hackers and malware developers are doing their best to crack into your phone and mess it up or steal all the data. And with the new technologies being used in modern smartphones, this has become extremely easy. So today we will take a look at some of the top antivirus software you can use on your iPhone for better security. Read on to find out more. McAfee Mobile Security McAfee is considered as the perfect security tool for your iPhone if you want to keep nosy family members and friends away from the...

IoT, its future and its impact on our lives

A radical change in our lives brought about by the Internet of Things – An overview

How to get your business through stormy weather

Having your own business is very rewarding in many ways, but it comes with a price. When you run your own business, no matter how big or small, you are responsible for yourself and the people that you employ, there is no monthly paycheck unless you provide for it. That is why having a solid financial base is crucial to keep your business alive if or when the going gets rough. There are lots of reasons your income or turnover could slack, not the right season, the economy is slow, there is a new and better product on the market or even new competition. In most cases, if you play your cards...