I have been working with DW/BI Technologies for over 12 years now. Of late the hype seems to be around Big Data Analytics and the technologies around that. While some of the hype is understandable for a new technology a look around the technology landscape, you immediately know these technologies are nowhere near ready for mass consumption. One of the main reasons for this of course is, not everybody has a big data problem. Still many of the problems that we deal with, even at the enterprise level are adequately addressed by databases and DW technology that has existed for a long time.
All these technologies are relatively immature and most of the time not usable once you get past the developer community. Forget about your end users getting comfortable with these technologies it's not going to happen, not at the present level of maturity. One of the other things about hadoop and hbase is that, it seems like they are all open source and you are getting it all free or for very low cost. But scalability always comes at a cost.
Company i work for is into Big Data Technologies in a big way and don't get me wrong, i spend an awful lot of time more than i would like fiddling around with these technologies because its interesting. But if you are going to make an investment on big data analytics then i think the first question has to be will it produce actionable insights. This problem is not unique to Big Data projects. According to several research papers 80% of DW/BI projects are a failure because it doesn't translate into business benefits.
DW/BI projects are unique because to make it work, it requires complete end user and stakeholder involvement. Many departments in most companies will have internal dynamics, so there will be problems in enforcing data uniformity, Master Data Management and agreeing on key metrics that is important for business. What this does most of the times, if there is no strong leadership will take the BI project in multiple directions resulting in a free for all.
With Big Data, more data is going to collected from all these various divisions, which then needs to be analyzed and results derived out of using reporting. So in my opinion, any company setting out on big data analytics has to have a very good view of what their end result is, to avoid getting lost in the sea of data.
As developers and architects, one other important thing to remember suitability of a technology for a particular problem. Just because its new and hip, big data analytics technologies doesn't always work for all problems. Traditional databases and BI as it's done now will still apply to a wide range of problems.
So if you ask my opinion, i would say, get yourself familiar with hadoop and hbase etc. and if you get a problem to work on with these technologies its great. Stay up with these technologies, but recognize they are not for every problem. Last but not the least, prepare yourself by enhancing your knowledge on how databases deal with some of the problems of scalability and high availability that hadoop and big data technologies is supposed to address.
All these technologies are relatively immature and most of the time not usable once you get past the developer community. Forget about your end users getting comfortable with these technologies it's not going to happen, not at the present level of maturity. One of the other things about hadoop and hbase is that, it seems like they are all open source and you are getting it all free or for very low cost. But scalability always comes at a cost.
Company i work for is into Big Data Technologies in a big way and don't get me wrong, i spend an awful lot of time more than i would like fiddling around with these technologies because its interesting. But if you are going to make an investment on big data analytics then i think the first question has to be will it produce actionable insights. This problem is not unique to Big Data projects. According to several research papers 80% of DW/BI projects are a failure because it doesn't translate into business benefits.
DW/BI projects are unique because to make it work, it requires complete end user and stakeholder involvement. Many departments in most companies will have internal dynamics, so there will be problems in enforcing data uniformity, Master Data Management and agreeing on key metrics that is important for business. What this does most of the times, if there is no strong leadership will take the BI project in multiple directions resulting in a free for all.
With Big Data, more data is going to collected from all these various divisions, which then needs to be analyzed and results derived out of using reporting. So in my opinion, any company setting out on big data analytics has to have a very good view of what their end result is, to avoid getting lost in the sea of data.
As developers and architects, one other important thing to remember suitability of a technology for a particular problem. Just because its new and hip, big data analytics technologies doesn't always work for all problems. Traditional databases and BI as it's done now will still apply to a wide range of problems.
So if you ask my opinion, i would say, get yourself familiar with hadoop and hbase etc. and if you get a problem to work on with these technologies its great. Stay up with these technologies, but recognize they are not for every problem. Last but not the least, prepare yourself by enhancing your knowledge on how databases deal with some of the problems of scalability and high availability that hadoop and big data technologies is supposed to address.
No comments:
Post a Comment