Nearly 50% of Big Data practitioners replying to a recent Wikibon survey report disappointing results from their Big Data projects, and an unfortunately 2% report total failures, writes Wikibon Big Data Analyst Jeff Kelly. While companies generally expect $3-$4 of business value for every $1 of investment in Big Data systems, they are actually achieving $0.55 of return.

Kelly attributes these partial or, in some cases, complete failures to three causes:

1. Insufficient Planning: Big Data projects initiated by IT departments in particular often lack compelling business use cases and end up as underutilized technical exercises.
2. Lack of Skilled Big Data Practitioners: Data Scientists and technical experts in the new technologies — Hadoop, Map Reduce, and NoSQL databases like MarkLogic — are rare on the ground and hard to recruit.
3. ‘Raw” and Immature Technologies: Many of the technologies available today are user unfriendly at best and difficult even for IT professionals to manage adequately. The general low staffing levels of many IT departments, a product of the post-2008 economy, does not help.

As a result, Kelly strongly recommends that Big Data projects originate not in the IT department but in business groups such as marketing and start with specific definitions of the business value to be derived. Then he recommends that users either hire consultants with strong Big Data skills and experience or use Big-Data-as-a-Service offerings rather than building the system in-house.

Jeff Kelly’s full analysis  is available without charge on the Wikibon Web Site. IT professionals are invited to register for free membership in the Wikibon Community. Members can post comments on published research and publish their own questions, tips, Professional Alerts, and white papers on the site. They also receive invitations to the periodic Peer Insight Meetings, where their peers discuss how they are solving business and technical problems with creative use of advanced technologies.