March 2, 2014
Posted by Andrei Pandre under Cloud
, Guest Posts
Leave a Comment
For this weekend I got 2 guest bloggers (one yesterday and other today) sharing their thoughts about Cloud Services for BI and DV. I myself published recently a few articles about this topic, for example here: http://apandre.wordpress.com/2013/08/28/visualization-as-a-service/ and here:
http://apandre.wordpress.com/2013/12/14/spotfire-cloud-pricing/ . My opinions can be different from Guest Bloggers. You can find many providers of DV and BI Cloud Services, including Spotfire Cloud, Tableau Online, GoodData, Microstrategy Cloud, Bime, Yellofin, BellaDati, SpreadsheetWEB etc.
Let me introduce my 2nd guest blogger for this weekend: Ugur Kadakal is the CEO and founder of Pagos, Inc. located in Cambridge, MA. Pagos is the developer of SpreadsheetWEB which transforms Excel spreadsheets into web based Business Intelligence (BI) applications without any programming. SpreadsheetWEB can also convert PowerPivot files into web based dashboards. It provides advanced Data Visualization (DV) to SQL Analysis Services (Tabular) cubes without SharePoint. Mr. Kadakal published a few articles on this blog before with great feedback, so he is a serial Guest Blogger.
Traditional BI versus Cloud BI
Over the past several years, we have been witnessing numerous transformations in the software industry, from a traditional on-premise deployment model to the Cloud. There are some application types for which cloud makes a lot of sense while it doesn’t for some others. BI is somewhere in between.
Before I express my opinion on the subject of Traditional BI versus Cloud BI, I would like to clarify my definitions. I define traditional BI as large enterprise implementations which connect with many data sources in real-time. These projects have many phases and require large teams to implement. These projects could take years and cost millions of dollars to implement.
Many people define cloud BI as deployments on a proprietary, third-party, multi-tenant environment managed by a vendor. My definition is somewhat different and broader. Cloud BI is more about ease of deployment, use and management. While Cloud BI can be hosted and managed by a vendor, it can also be deployed on a private Cloud infrastructure like Amazon or Microsoft Azure. With the advancement of cloud infrastructure technologies like OpenStack, deploying and managing private cloud infrastructure is becoming easier for many enterprises. As a result, whether Cloud BI is deployed on a multi/single-tenant environment on vendor infrastructure, a third party cloud infrastructure like Amazon, Azure, etc. or on internal private cloud, it becomes more of a business decision rather than a technical limitation.
One main distinction between Traditional BI and Cloud BI is data management. Traditional BI implementations can have real-time data as they can connect to the original data sources directly. I don’t believe that Cloud BI should deal with real-time data, even if implemented on internal private cloud infrastructure. Supporting real-time data is a requirement that makes any BI project complicated and costly. Hence Cloud BI solutions should include simple utilities i.e. ETL, residing on local computers to push internal data into Cloud BI’s data model periodically. Since Cloud BI should not deal with real-time data scenarios, this data synchronization can be configured by the business user accordingly.
Another distinction is the ease of implementation. Regardless of where it is deployed, Cloud BI solutions should take no more than a few hours to implement and configure. Some BI vendors already support images on Amazon cloud to simplify this process.
Traditional BI model typically requires significant upfront investments. Part of this investment is internal while the rest is BI licensing and implementation fees. But the very nature of Cloud BI requires agility from deployment to data management and dashboard creation. Cloud BI project can be deployed easily and it can also be modified and shut down with equal ease. Hence traditional business model of large upfront investments doesn’t make sense here. Cloud BI business model should be subscription based regardless of whether it is implemented on a vendor infrastructure or on an on-premise private cloud infrastructure. Customers should be able to pay what they use and for how long they use it. Such simplicity will also eliminate vendor lock-in risks that most enterprises have to mitigate.
In summary, there are many BI projects that will require traditional BI implementation. These projects typically require real-time data and connectivity to many different data sources. Cloud BI should not attempt to handle these types of projects. But there are many other BI projects that require neither real-time data nor the data which comes from different systems that should be connected. Cloud BI can handle these projects quickly and cost effectively, by empowering business users to manage the whole process without IT or external support. From discovery to data synchronization to dashboard creation and management, every activity can be handled by business users.
March 1, 2014
Posted by Andrei Pandre under Cloud
, Guest Posts
For this weekend I got 2 guest bloggers (one today and second tomorrow) sharing their thoughts about Cloud Services for BI and DV. I myself published recently a few articles about this topic, for example here: http://apandre.wordpress.com/2013/08/28/visualization-as-a-service/ and here:
http://apandre.wordpress.com/2013/12/14/spotfire-cloud-pricing/ . My opinions can be different from Guest Bloggers (see my comment below this article). You can find many providers of DV and BI Cloud Services, including Spotfire Cloud, Tableau Online, GoodData, Microstrategy Cloud, Bime, Yellofin, BellaDati, SpreadsheetWEB etc.
Let me introduce my 1st guest blogger for this weekend: Mark Flaherty is Chief Marketing Officer at InetSoft Technology, a BI (Business Intelligence) software provider founded in 1996, headquartered in Piscataway, New Jersey with over 150 employees worldwide. InetSoft’s flagship BI application Style Intelligence enables self-service BI spanning dashboarding, reporting and visual analysis for enterprises and technology providers. The server-based application includes a data mashup engine for combining data from almost any data source and browser-based design tools that power users and developers can use to quickly create interactive DV (Data Visualizations).
Are public BI cloud services really going to overtake the traditional on-premise deployment of BI tools?
(Author: Mark Flaherty. Text below contains Mark’s opinions and they can be different from opinions expressed on this blog).
It’s been six years since public BI cloud services came to be. Originally termed SaaS BI, public BI cloud services refers to commercial service providers who host a BI application in the public cloud that accesses corporate data housed in the corporate private cloud and/or other application providers’ networks. As recently as last month, an industry report from TechNavio said, “the traditional on-premise deployment of BI tools is slowly being taken over by single and multi-tenant hosted SaaS.” I have a feeling this is another one of those projections that copies a historical growth rate forward for the next five years. If you do that with any new offering that starts from zero, you will always project it to dominate a marketplace, right?
I thought it would be interesting to discuss why I think this won’t happen.
In general, there is one legitimate driving force for why companies look to cloud solutions that helps drive the demand for cloud BI services specifically: outsourcing of IT. The types of companies for whom this makes the most sense are small businesses. They have little or no IT staff to set up and support enterprise software, and they also have limited cap-ex budgets so software rentals fit their cash flow structure better. While this is where most of the success for cloud BI has happened, this is only a market segment opportunity. By no means do small companies dominate the IT marketplace.
Another factor for turning to public cloud solutions is expediency. Even at large companies where there is budget for software purchases, the Business sometimes becomes frustrated with the responsiveness of internal IT, and they look outside for a faster solution. This makes sense for domain-specific cases where there is a somewhat narrow scope of need, and the application and the data are self-contained. Salesforce.com is the poster child for this case, where it can quickly be set up as a CRM for a sales team. Indeed the fast success of salesforce.com is a big reason why people think cloud solutions will take off in every domain.
But business intelligence is different. A BI tool is meant to span multiple information areas, from finance to sales to support and more. This is where it gets complicated for mid-sized and global enterprises. The expediency factor is nullified because the data that business users want to access with their cloud BI tool is controlled by IT, so they need to be involved. Depending on the organization’s policies and politics, this can either slow down such a move or kill it.
The very valid reason why enterprise IT would kill the idea for a public cloud BI solution is why ultimately I think public BI cloud services has such a limited opportunity in the overall market. One of IT’s responsibilities is ensuring data security, and they will rightly point out the security risks of opening access to sensitive corporate data to a 3rd party. It’s one thing to trust a vendor with one set of data like website visitor traffic, but trusting them with all of a company’s financial and customer data is where almost all companies will draw the line. This is a concern I don’t see ever going away.
What are some pieces of evidence that public BI cloud services have a limited market opportunity? When BI cloud services first came onto the scene, all of the big BI vendors dabbled in it. Now many no longer champion these hosted offerings, or they have shuttered or demoted them. IBM’s Cognos Express is now only an on-premise option. SAP BusinessObjects BI OnDemand can’t be found from SAP’s main site, but has its own micro site. Tibco’s Spotfire Cloud and Tableau Software’s Tableau Online are two exceptions among the better known BI providers that are still prominently marketed. However, Tibco positions this option for small businesses and workgroups and omits certain functionality.
Our company, too, experimented with a public BI cloud offering years ago. It was first targeted at salesforce.com customers who would want to mash up their CRM data with other enterprise-housed data. We found mostly small, budget challenged companies in their customer base, and the few large enterprises that we found balked at the idea, asking instead, for our software to be installed on-premise where they would connect to any cloud-hosted data on their own. Today the only remaining cloud offering of ours is a free visualization service called Visualize Free which is similar to Tableau Public or IBM’s Many Eyes.
Another observation to make, while there have been a handful of pure-play cloud BI vendors, one named “Lucidera,” came and went quite quickly. Birst is one that seems to have got a successful formula.
In summary, yes, there is a place for public BI cloud services in the small business market, but no, it’s not going to overtake traditional on-premise BI.
February 23, 2014
Posted by Andrei Pandre under Analytics
For last 6 years every and each February my inbox was bombarded by messages from colleagues, friends and visitors to this blog, containing references, quotes and PDFs to Gartner’s Magic Quadrant (MQ) for Business Intelligence (BI) and Analytics Platforms, latest can be found here: http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb .
Last year I was able to ignore these noises (funny enough I was busy by migrating thousands of users from Business Objects and Microstrategy to Tableau-based Visual Reports for very large company), but in February 2014 I got so many questions about it, that I am basically forced to share my opinion about it.
1st of all, as I said on this blog many times that BI is dead and it replaced by Data Visualization and Visual Analytics. That was finally acknowledged by Gartner itself, by placing Tableau, QLIK and Spotfire in “Leaders Quarter” of MQ for 2nd year in a row.
2ndly last 6 MQs (2009-2014) are suspicious for me because in all of them Gartner (with complete disregard of reality) placed all 6 “Misleading” vendors (IBM, SAP, Oracle, SAS, Microstrategy and Microsoft) of wasteful BI platforms in Leaders Quarter! Those 6 vendors convinced customers to buy (over period of last 6 years) their BI software for over $60B plus much more than that was spent on maintenance, support, development, consulting, upgrades and other IT expenses.
There is nothing magic about these MQs: they are results of Gartner’s 2-dimensional understanding of BI, Analytics and Data Visualization (DV) Platforms, features and usage. 1st Measure (X axis) according to Gartner is the “Completeness of Vision” and 2nd Measure (Y axis) is the “Ability to Execute”, which allows to distribute DV and BI Vendors among 4 “Quarters”: RightTop – “Leaders”, LeftTop -”Challengers”, RightBottom – “Visionaires” and LeftBottom – “Niche Players” (or you can say LeftOvers).
I decided to compare my opinions (expressed on this blog many times) vs. Gartner’s (they wrote 78 pages about it!) by taking TOP 3 Leaders from Gartner, than taking 3 TOP Visionaries from Gartner (Projecting on Axis X all Vendors except TOP 3 Leaders) than taking 3 TOP Challengers from Gartner (Projecting on Axis Y all Vendors except TOP 3 Leaders and TOP 3 Visionaries ) than TOP 3 “Niche Players” from the Rest of Gartner’s List (above) and taking “similar” choices by myself (my list is wider then Gartner’s, because Gartner missed important to me DV Vendors like Visokio and vendors like Datawatch and Advizor Solutions are not included into MQ in order to please Gartner’s favorites), see the comparison of opinions below:
If you noticed, in order to be able to compare my opinion, I had to use Gartner’s terms like Leader, Challenger etc., which is not exactly how I see it. Basically my opinion overlapping with Gartner’s only in 25% of cases in 2014, which is slightly higher then in previous years – I guess success of Tableau and QLIK is a reason for that.
BI Market in 2013 reached $14B and at least $1B of it spent on Data Visualization tools. Here is the short Summary of the state of each Vendor, mentioned above in “DV Blog” column:
Tableau: $232M in Sales, $6B MarketCap, YoY 82% (fastest in DV market), Leader in DV Mindshare, declared goal is “Data to the People” and the ease of use.
QLIK: $470M in Sales, $2.5B MarketCap, Leader in DV Marketshare, attempts to improve BI, but will remove Qlikview Desktop from Qlik.Next.
Spotfire: sales under $200M, has the most mature Platform for Visual Analytics, the best DV Cloud Services. Spotfire is limited by corporate Parent (TIBCO).
Visokio: private DV Vendor with limited marketing and sales but has one of the richest and mature DV functionality.
SAS: has the most advanced Analytics functionality (not easy to learn and use), targets Data Scientists and Power Users who can afford it instead of free R.
Revolution Analytics: as the provider of commercial version and commercial support of R library is a “cheap” alternative to SAS.
Microsoft: has the most advanced BI and DV technological stack for software developers but has no real DV Product and has no plan to have it in the future.
Datawatch: $33M in sales, $281M MarketCap, has mature DV, BI and real-time visualization functionality, experienced management and sales force.
Microstrategy: $576M in sales, 1.4B MarketCap; BI veteran with complete BI functionality; recently realized that BI Market is not growing and made the desperate attempt to get into DV market.
Panorama: BI Veteran with excellent easy to use front-end to Microsoft BI stack, has good DV functionality, social and collaborative BI features.
Advizor Solutions: private DV Veteran with almost complete set of DV features and ability to do Predictive Analytics interactively, visually and without coding.
RapidMiner: Commercial Provider of open-source-based and easy to use Advanced Analytical Platform, integrated with R.
In addition to differences mentioned in table above, I need to say that I do not see that Big Data is defined well enough to be mentioned 30 times in review of “BI and Analytical Platforms” and I do not see that Vendors mentioned by Gartner are ready for that, but may be it is a topic for different blogpost…
February 15, 2014
Posted by Andrei Pandre under Tableau
We were told (5+ month ago) what to expect from Tableau 8.2 (originally @TCC13 they said Release can be before the end of the winter of 2014; however in the latest Earnings Call here: http://seekingalpha.com/article/1994131-tableau-softwares-ceo-discusses-q4-2013-results-earnings-call-transcript CEO acknowledged the delay: 8.2 in Q2 of 2014, and v.9 in “first half of 2015″, many months later then original plan), including:
- Tableau for MAC (very timely at time when QLIK about to abandon the Qlikview Desktop in favor of HTML5 Client),
- Story Points (new type of worksheet/dashboard with mini-slides as story-points, so bye-bye to Powerpoint),
- seamless access to data via data connection interface to visually build a data schema, including inner/left/right/outer joins,
- ability to beautify the columns names.
I am sure Tableau already has a Roadmap for Tableau 9 and beyond, but I accumulated a list of wishes for it (may be it is not too late to include some of it to Roadmap?). This Wishlist is rather about backend than about front-end Eye Candies (the nature of the Large Enterprise dictates that). Here it is:
- Visual ETL functionality and Data Quality Validation/Cleaning;
- (thanks to Larry Keller): Enterprise Repository for pre-Validated Sharable Regularly Refreshed Data Extracts, Data Connections and Data Sources;
- Ability to collect Data automatically (say Machine-generated or/and transactional Data) and Visually (say from Humans, filling Data-Entry Forms), both tied to already predefined and/or modifiable Data Extracts;
- Visual Data Modeling;
- Real-Time Visualization, support (Spotfire and Datawatch have it!) for Complex Event Processing (CEP), Visual Alerts and Alarms;
- Scripting for Visual Predictive Modeling and Visual Data Mining with ability to do it in Visual IDE and minimal Coding;
- Better integration with R (current integration is limited to 4 functions passing parameters to R Server), with Visual IDE and minimal or NO Coding.
- Enterprise-wide source control and change management.
- Please allow to share Data Visualizations (read-only) from Tableau Online for free (learn from Spotfire Cloud, it called Public Folder!), otherwise it will be too much of usage of free Tableau Reader. Currently, in order to access to published on Tableau Online workbooks Tableau by default requiring the extra subscription, which is wrong from my point of view, because you can just publish it on Public Folder of such site (similar to what Spotfire Cloud does). By default Tableau Online does not allow the usage of Public Folder, which contradicts the spirit of Tableau Reader and creates unnecessary negative feeling toward Tableau.
- Enterprise-wide reuse of workbooks and visual designs etc.
Since Tableau is going into enterprise full speed (money talks?) then it needs to justify its pricing for Tableau Server, especially if Tableau wish to stay there for long. Feel free to add to this list (use comments or email for it). The first addition I got in a few hours after posting the Wishlist above from Mr. Damien Lesage, see 3 additions from Damien below and his entire comment below of this blogpost:
- Tableau Server for Linux (I actually advocated it for a while since Microsoft changed (made CALs more expensive, now it looks to me as unwarranted taxation) its Client Access Licensing for Window Server 2012). For comparison Spotfire Server for Linux and Solaris existed for years: http://support.spotfire.com/sr_spotfireserver60.asp , and it is one of reasons why large enterprises may choose Spotfire over Tableau or Qlikview;
- Extra visualization capability: hierarchical, network and graph representations of data (do we need an approval of Stephen Few for that?);
- Ability for extract engine to distribute extracts between different servers to allow to load them more quickly and support bigger datasets (I suggest additional ability to do it on workstations too, especially with Tableau Desktops installed and it means they have TABLEAU.COM executable installed anyway)
Suggestion from Mike Borner (see his comment below):
- ability to report metadata/calculated fields
Now I can extend my best wishes for you onto 2015 due the delay of Tableau 9!
February 11, 2014
Tableau Software (symbol DATA) did something that nobody or almost nobody in BI and/or Data Visualization (DV) field did before with this or larger size of Revenue. Tableau in their last Quarter of 2013 Fiscal Year (reported last week) increased their Year-over-Year Ratio for both Quarterly accounting (95%) and Yearly accounting (82%, way above all DV and BI competitors) while dramatically increased their Revenue to $232M per Year, see it here: http://investors.tableausoftware.com/investor-news/investor-news-details/2014/Tableau-Announces-Fourth-Quarter-and-Full-Year-2013-Financial-Results/default.aspx.
You can compare on diagram below the growth of 3 competitors over last 6 years (2008-2013, Spotfire sales unavailable since TIBCO (symbol TIBX) bought it): BI veteran Microstrategy (bluish line slowing down last 2+ years), largest DV vendor Qliktech (symbol QLIK, red line, decreasing Year-over-Year growth) and fastest growing DV Vendor Tableau (yellow line with Record Year-over-Year growth):
Tableau stock was and is overpriced since its IPO (e.g. today EPS is -0.19 and P/E ratio is very high, see it here: http://ycharts.com/companies/DATA/pe_ratio). If you follow Warren Buffet (Buy Low, Sell High), today is a good day to sell a DATA stock, unless you intend to hold it for long or forever. However many people ignore Warren and volume of buying for last few days was above average (780K for DATA) and above 1 million shares per day (e.g. on 2/5/14 it was 4.4M of shares). On OpenInsider you can find at least 2 people, who agreed with Warren and sold during last few days 700000 Tableau’s shares for total $62M+ (guess who it can be? Chris and Christian – part of 1% since 5/17/13 IPO…):
As the result, the $DATA (Tableau’s Symbol) jumped up $10+ from already overvalued share price to $97+ after 2/14/14, today it added $5 (click on image below to enlarge it) to share price and keeps going up:
BY end of 2/14/14 Tableau’s Market Capitalization went over $5.96B, twice more then Qliktech’s MarketCap (which is almost the same as a year ago) and $2B more then TIBCO’s MarketCap (which is almost the same as a year ago)! Basically, Tableau’s MarketCap as of end of trading day today is almost the same as combined MarketCap of QLIK and TIBX.
For me the more important indicator of company’s growth is a “HRI” (Hiring Rate Indicator as the ratio of the number of open positions to the number of Full-Time employees of the company). As of today, Tableau has 216 job openings (current estimate is has about 1100 employees), Qliktech has 101 openings (while employed 1700 people) and Spotfire has about 34 open positions (current estimate of number of Spotfire Employees is difficult because it is completely inside TIBCO, but probably still below 500). It means that Tableau’s HRI is 19.6%, Qliktech’s HRI is 5.9% and Spotfire’s HRI is below 6.8%.
February 1, 2014
Analytics extrapolates Visible Data to the future (“predicts”) and enables us to see more then 6-dimensional subsets of data with mathematical modeling. The ability to do it visually, interactively and without programming … vastly expands the number of potential users for Visual Analytics. I am honored to present the one of the most advanced experts in this area – Mr. Gogswell: he decided to share his thoughts and be the guest blogger here. So the guest-blog-post below is written by Mr. Douglas Cogswell, the Founder, President and CEO of ADVIZOR Solutions Inc.
Formed in 2003, ADVIZOR combines data visualization and in-memory-data-management expertise with usability knowledge and predictive analytics to produce an easy to use, point and click product suite for business analysis. ADVIZOR’s Visual Discovery™ software spun out of a distinguished research heritage at Bell Labs that spans nearly two decades and produced over 20 patents.
Mr. Cogswell is the well known thought leader and he is discussing below the next step in Data Visualization Technology, when limitation of human eye prevents users to comprehend the multidimensional (say more than 6 dimensions) Data Patterns or estimate/predict the future trends with Data from the Past. Such Multidimensional “Comprehension” and Estimations of the Future Trends requires a Mathematical Modeling in form of Predictive Analytics as the natural extension of Data Visualization. This is in turn, requires the Integration of Predictive Analytics and Interactive Data Visualization. Such Integration will be accepted much easier by business and analysts , if it will require no coding.
Mr. Cogswell discussing the need and possibility of that in his article (Copyright ADVIZOR Solutions, 2014) below.
Integrating Predictive Analytics and Interactive Data Visualization WITHOUT any Coding!
It’s a new year, and many organizations are mulling over how and where they will make new investments. One area getting a lot of attention these days is predictive analytics tools. The need to better understand the present and predict what might happen in the future for competitive advantage is enticing many to look at what these tools can do. TechRadar spoke with James Fisher, who said 85 percent of the organizations that have adopted these tools believe they have positively impacted their business.
Fast Fact Based Decision Making is Critical.
“Businesses are collecting information on their customers’ mobile habits, buying habits, web-browsing habits… The list really does go on,” he said. “However, it is what businesses do with that data that counts. Analytics technology allows organizations to analyze their customer data and turn it into actionable insights, in a way that benefits business.”
Interest in predictive analytics by businesses is expected to continue to grow well beyond this year, with Gartner reporting in early 2013 that approximately 70 percent of the best performing enterprises will either manage or have a view of their processes with predictive analytics tools by 2016. By doing this, businesses will gain a better sense of what is happening within their own networks and corporate walls, which actions could have the best impact and give increased visibility across their industries. This will give situational awareness across the business, making operating much easier than it has been in past years.
Simplicity and Ease of Use are Key.
Analytics is something every business should be figuring out. There are more software options than ever, so executives will need to figure out which solution will work best for them and their teams. According to InformationWeek’s Doug Henschen, the “2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey” found that business users and salespeople need easy-to-use, visual data analytics that is intuitive and easily accessible from anywhere, any time. . These data visualization business intelligence tools can give a competitive edge to the companies adopting them.
“The demand for these more visual analytics tools leads to one of the biggest complaints about analytics,” he said. Ease-of-use challenges have crippled the utilization rate of this software. But that is changing. “Analytics and BI vendors know that IT groups are overwhelmed with requests for new data sources and new dimensions of data that require changes to reports and dashboards or, worse, changes to applications and data warehouses,” he wrote. “It’s no wonder that ‘self-service’ capabilities seem to be showing up in every BI software upgrade.”
A recent TDWI research report titled “Data Visualization and Discovery for Better Business Decisions” found that companies do have their future plans focused on these analytics and how they can use them. In fact, 60 percent said their organizations are currently using business visualization for snapshot reports, scorecards, or display. About one-third are using it for discovery and analysis and 26 percent for operational alerting. However, companies are looking to expand how they use the technology, as 45 percent are looking to adopt it for discovery and analysis, and 39 percent for alerts.
“Visualization is exciting, but organizations have to avoid the impulse to clutter users’ screens with nothing more than confusing ‘eye candy’,” Stodder wrote. “One important way to do this is to evaluate closely who needs what kind of visualizations. Not all users may need interactive, self-directed visual discovery and analysis; not all need real-time operational alerting.”
Data Visualization & Predictive Analytics Naturally Complement Each Other.
Effective data visualizations are designed to complement human perception and our innate ability to see and respond to patterns. We are wired as humans to perceive meaningful patterns, structure, and outliers in what we see. This is critical to making smarter decisions and improving productivity, and essential to the broader trend towards self-directed analysis and BI reporting, and tapping into new sources of data.
Visualization also encourages “storytelling” and new forms of collaboration. It makes it really easy to not only “see” stories in data, but also to highlight what is actionable to colleagues.
On the other hand, the human mind is limited in its ability to “see” very many correlations at once. While visualization is great for seeing patterns across 2, or 4 or maybe 6 criteria at a time, it breaks down when there are many more variables than that. Very few people are able to untangle correlations and patterns across, say, 15 or 25 or 75 or in some cases 300+ criteria that exist in many corporate datasets.
Predictive Analytics, on the other hand, is not capacity constrained!! It uses mathematical tools and statistical algorithms to examine and determine patterns in one set of data . . . in order to predict behavior in another set of data. It integrates well with in-memory-data and data visualization, and leads to faster and better decision making.
Making it Simple & Delivering Results.
The challenge is that most of the predictive analytics software tools on the market require the end-user to be able to program in SQL in order to prep data, and have some amount of statistics background to build models in R … or SPSS … or SAS. At ADVIZOR Solutions our vision has been to empower business analysts and users to build predictive models without any code or statistics background.
The results have been extremely promising — inquisitive and curious-minded end-users with a sense for causality in their data can easily do this — and are turning around models in just a few hours. The result is they are using data in new and powerful ways to make better business decisions.
Three Key Enablers to a Simple End-User Process.
The three keys to making this happen are: (1) having all the relevant data offloaded from the database or datamart into RAM, (2) allowing the business user to explore it visually, and (3) providing a really simple modeling interface.
Putting the data in RAM is key to making it easy to condition so that the business user can create modeling factors (such as time lags, factors from data in multiple tables, etc.) without having to go back and condition data in the underlying databases — which is usually a time consuming process that involves coordinating with IT and/or DBAs.
Allowing the business user to explore it visually is key to hypothesis generation and vetting about what really matters, before building and running models.
Providing really simple interfaces that automate the actual statistics part of the process lets the business user focus on their data, not the statistics of the model. That simple modeling process includes:
- Select the Target & Base Populations
- The “target” is the group you want to study (e.g., people who responded to your campaign)
- The “base” is the group you want to compare the target to (e.g., everybody who received the campaign)
- Visually Explore the data and develop Hypotheses
- This helps set up which explanatory fields to include …
- … and which additional ones may need to be added
- Select list of Explanatory Fields
- The “explanatory fields” are the factors in your data that might explain what makes the target different from other entities in your data
- Build Model
- Understand and Communicate what the model is telling you
- Predict / Score Base Population
- Get lists of Scored potential targets
Check out how you can do this with no code in this 8 min YouTube video.
Best Done In-house with Your Team.
In our experience this type of work is best done in-house with your team. That’s because it’s not a “black box”, it’s a process. And since your team knows the data and its context better than anybody else, they are the ones best suited to discuss, interpret, and apply the results. In our experience, over and over again it has been proven that knowing the data and context is the key factor … and that you don’t need a statistics degree to do this.
Quick Example: Consumer Packaged Goods Sales.
In recent client work a well known consumer packaged goods company was trying to untangle what was driving sales. They had several key questions they were attempting to answer:
- What factors drive sales?
- How do peaks in incremental sales relate to the Social Media spikes?
- For all brands
- By each brand
- How does it vary by media provider? By type of post?
- Can we use this data to forecast incremental sales? Which factors have the biggest impact?
They had lots of data, which included sales by brand by week, and a variety of potential influences which included: a variety of their own promotions, call center stats, social media posts, and mined sentiment from those social media posts (e.g., was the post “positive”, “neutral”, or “negative”). The key step in creating the right explanatory fields was developing time lags for each of these potential influences since the impact on sales was not necessarily immediate — for example, positive Twitter posts this week may have some impact on sales, but more likely the impact will be on sales +1 week, or maybe +2 weeks, or +4 weeks, etc.
What we learned was that there were multiple influences and their intensity varied by brand. Seasonality was no longer the major driver. New influences — including social media posts and online promotions — were now in the top spot. We also learned that the key influences can and should be managed. This was critical — there are lags between the impact of, for example, a negative Twitter post and when it hits sales. As a result, a quick positive response to a negative post can heavily offset that negative post.
An easy to use data discovery and analysis tool that integrates predictive analytics with interactive data visualization and which is then placed in the hands of business analysts and end-users can make huge differences in how data is analyzed, how fast that can happen, and how it is then communicate to and accepted by the decision makers in an organization.
And, stay tuned. We’ll next be talking about the people side of predictive analytics — if there is now technology that lets you create and use models without writing any code, then what are the people skills and processes required to do this well?
January 19, 2014
This is a repost from Data Visualization Consulting Page.
Visitors of this blog generated a lot of requests for my Data Visualization “Advice” (small projects for a few hours or days, no NDA [Non-Disclosure Agreement] involved), for Data Visualization Consulting projects (a few weeks or months; I tend to avoid the NDAs as they can interfere with my blogging activities) and even for Full-time work (for example my latest full-time job I got because my employer often visited and read my blog; NDA needed).
Additionally, sometimes I am doing free-of-charge work, if involved projects are short, extremely interesting for me and beneficial for my Data Visualization Blog, like this project:
Obviously all these projects can be done only when I have spare time either from full-time work and/or other projects, duties and activities.
I also cannot relocate or travel, so I can do it mostly from my home office – telecommuting (RDP, Skype, phone, WebEx, GoToMeeting etc.) or if client is local to Massachusetts, then sometime I can visit Client’s site, see below the Map of my Local “Service Area” – part of Middlesex County between Routes 495, 3 and 20 – where I can commute to Client’s Location (please click on map below to enlarge the image) :
If I do have time for short-term advisory projects (from 2 hours to 2 weeks), clients usually pay by the highest rate, similar to what Qliktech, Spotfire, Tableau or IBM charging for their Consulting Services (I consider my consulting as better service than theirs…). If you will go to this thread on Tableau Community:
http://community.tableausoftware.com/thread/127338 then you will find these Indicative Rates for Consulting Tableau Work (Qlikview and Spotfire Rates are very similar):
Low $125, Max $300, Average around $175 per hour.
Here are the most popular requests for my Advisory work:
- Visual Design and Architectural Advice for Monitoring or Operational Dashboard(s);
- Review of Data Visualization Work done by my Clients;
- Prototyping of Data Visualizations (most requested by my visitors);
- My opinion on Strengths and Weaknesses of Data Visualization Vendor/Product, requested by trader, portfolio or hedge fund manager(s)
- Advice about what Hardware to buy (say to get the most from Tableau License client has);
- Advice what Charts and Filters to use for given Dataset and Business Logic;
- Technical Due Diligence on Data Visualization Startup for Venture Capitalists investing into that Start-up.
For mid-size projects (from 2 weeks to 6 months) clients getting a “Progressive” discount – the longer the project then the larger the discount. Here are the most popular requests for my Consulting Data Visualization Work:
- Comparing Data Visualization Product vs. Other Visualization Product for specific Client’s needs and projects;
- Comparing Clients’s Visualization Product vs. Competitor(s) Visualization Product (most requested);
- Benchmarking one or more Visualization Product(s) vs. specific data and application logic.
- Managed Clients migration of their Reporting and Analytical IT Infrastructure from obsolete BI Platforms like Business Objects, Cognos and Microstrategy to modern Data Visualization Environments like Tableau, Qlikview and Spotfire.
Full-time work (1 year or more engagements) is not exactly a Consulting but Full-time job when clients asking me to join their company. These jobs are similar to what I had in the past: Director of Visual Analytics, Data Visualization Director, VP of Data Visualization, Principal Data Visualization Consultant, Tableau Architect etc. Here are samples of full-time projects:
- Created, Maintained and Managed the Data Visualization Consulting Practices for my company/employer;
- Led the growth of Data Visualization Community (the latest example – 4000 strong Tableau Community) with own Blog, Portal and User Group behind the corporate firewall, created Dozens of near-real-time Monitoring Dashboards for Analytical and Data Visualization Communities;
- Designed and Implemented myself hundreds of Practical Data Visualizations and Visual Reports, which led to discovery of trends, outliers, clusters and other Data Patterns, Insights and Actions;
- Created hundreds of Demos, Prototypes and Presentations for Business Users;
- Designed Data Visualization Architecture and Best Practices for Dozen of Analytical Projects;
- Significantly improved the Mindshare and increased the Web Traffic to website of my company, Created and Maintained the Data Visualization blog for it.
You can find more observations about relationship between Full-Time salary and Hourly Rate for consulting in my previous post (from 6 months ago) here: http://apandre.wordpress.com/2013/07/11/contractors-rate/