BI News


Last month Tableau and Qliktech both declared that Traditional BI is too slow (I am saying this for many years) for development and their new Data Visualization (DV software) is going to replace it. Quote from Tableau’s CEO: Christian Chabot: “Traditional BI software is obsolete and dying and this is very direct challenge and threat to BI vendors: your (BI that is) time is over and now it is time for Tableau.” Similar quote from Anthony Deighton, Qliktech’s CTO & Senior VP, Products: “More and more customers are looking at QlikView not just to supplement traditional BI, but to replace it“.

One of my clients – large corporation (obviously cannot say the name of it due NDA) asked me to advise of what to choose between Traditional BI tools with long Development Cycle (like Cognos, Business Objects or Microstrategy), modern BI tools (like JavaScript and D3 toolkit) which is attempt to modernize traditional BI but still having  sizable development time and leading Data Visualization tools with minimal development time (like Tableau, Qlikview or Spotfire).

Since main criterias for client were

  • minimize IT personnel involved and increase its productivity;

  • minimize the off-shoring and outsourcing as it limits interactions with end users;

  • increase end users’s involvement, feedback and action discovery.

So I advised to client to take some typical Visual Report project from the most productive Traditional  BI Platform (Microstrategy), use its prepared Data and clone it with D3 and Tableau (using experts for both). Results in form of Development time in hours) I put below; all three projects include the same time (16 hours) for Data Preparation & ETL, the same time for Deployment (2 hours) and the same number (8) of Repeated Development Cycles (due 8 consecutive feedback from End Users):

DVvsD3vsBI

It is clear that Traditional BI requires too much time, that D3 tools just trying to prolongate old/dead BI traditions by modernizing and beautifying BI approach, so my client choose Tableau as a replacement for Microstrategy, Cognos, SAS and Business Objects and better option then D3 (which require smart developers and too much development). This movement to leading Data Visualization platforms is going on right now in most of corporate America, despite IT inertia and existing skillset. Basically it is the application of the simple known principle that “Faster is better then Shorter“, known in science as Fermat’s Principle of least time.

This changes made me wonder (again) if Gartner’s recent marketshare estimate and trends for Dead Horse sales (old traditional BI) will stay for long. Gartner estimates the size of BI market as $13B which is drastically different from TBR estimate ($30B).

BIDeadHorseTheoryTBR predicts that it will keep growing at least until 2018 with yearly rate 4% and BI Software Market to Exceed $40 Billion by 2018 (They estimate BI Market as $30B in 2012 and include more wider category of Business Analytics Software as opposed to strictly BI tools). I added estimates for Microstrategy, Qliktech, Tableau and Spotfire to Gartner’s MarketShare estimates for 2012 here:

9Vendors

However, when Forrester asked people what BI Tools they used, it’s survey results were very different from Gartner’s estimate of “market share:

BIToolsInUse

“Traditional BI is like a pencil with a brick attached to it” said Chris Stolte at recent TCC13 conference and Qliktech said very similar in its recent announcement of Qlikview.Next. I expect TIBCO will say similar about upcoming new release of Spotfire (next week at TUCON 2013 conference in Las Vegas?)

Tableau_brick2

These bold predictions by leading Data Visualization vendors are just simple application of Fermat’s Principle of Least Time: this principle stated that the path taken between two points by a ray of light (or development path in our context) is the path that can be traversed in the least time.

Pierre_de_Fermat2Fermat’s principle can be easily applied to “PATH” estimates to multiple situations like in video below, where path from initial position of the Life Guard on beach to the Swimmer in Distress (Path through Sand, Shoreline and Water) explained: 

Even Ants following the Fermat’s Principle (as described in article at Public Library of Science here: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0059739 ) so my interpretation of this Law of Nature (“Faster is better then Shorter“) that  traditional BI is a dying horse and I advise everybody to obey the Laws of Nature.

AntsOn2SurfacesIf you like to watch another video about Fermat’s principle of Least Time and related Snell’s law, you can watch this: 
Google+

Qlikview 10 was released around 10/10/10, Qlikview 11 – around 11/11/11, so I expected Qlikview 12 to be released on 12/12/12. Qliktech press release said today that the next (after 11.2) version of Qlikview will be delivered under the new nickname Qlikview.Next in 2014 but “for  early adopter customers in a production environment in 2013”. I hope I can get my hands on it ASAP!

The new buzzword is Natural Analytics: “QlikView.Next’s key value as an alternative BI platform is in its use of Natural Analytics“. The new Qliktech motto that “Qlikview is a Replacement of Traditional BI” is similar to what we heard from Tableau leaders just 2 weeks ago on Tableau Customer Conference in Washington, DC.  Another themes I hear from Qliktech about Qliview.Next are sounds familiar too: Gorgeous, Genius, Visually Beautiful, Associative Experience, Comparative Analysis, Anticipatory, Drag and Drop Analytics.

Qlikview.Next will introduce “Data Dialogs” as live discussions between multiple users about Data they see and explore collectively, enabling “Social BI”. This reminds me the integration between TIBBR (TIBCO’s collaboration platform) and Spotfire, which existed since Spotfire 4.0.

Details about new features in Qlikview.Next will be released later, but at least we know now when Qlikview 12 (sorry, Qlikview.Next that is) will be available. Some features actually unveiled in generic terms::

  • Unified, Browser-Based HTML5 Client, which will automatically optimize itself for user’ device;

  • Automatic and Intelligent re-sizing of objects to fit user’s screen;

  • Server-side Analysis and Development, Web-based creation and delivery of content, Browser-based Development;

  • Data Storytelling, narrative and social with Data Dialogs;

  • Library and Repository for UI objects;

  • Multi-source Data Integration and new web-based scripting;

  • QlikView Expressor for advanced graphical Data Integration and Metadata Management;

  • Improved Data Discovery with associative experience across all the data, both in memory and on disks;

  • Open API: JSON, .NET SDK and as JavaScript API;

  • All UI Objects can be treated as extension Objects, customizable with their source files available to developers;

  • New Managment Console with Qlikview on Qlikview Monitor;

  • New visualization capabilities, based on advanced data visualization suite from NComVA (bought by Qliktech a few months ago), potential samples see here: http://www.ncomva.se/guide/?chapter=Visualizations

NComVAVisualizations11

In addition Qliktech is launching the “Qlik Customer Success Framework” , which includes:

  • Qonnect Partner Program: An extensive global network of 1500+ partners, including resellers, (OEMs), technology companies, and system integrators.

  • Qlik Community: An online community with nearly 100,000 members comprised of customers, partners, developers and enthusiasts.

  • Qlik Market: An online showcase of applications, extensions and connectors.

  • Qoncierge: A single point of contact service offering for customers to help them access the resources they need.

  • Comprehensive Services: A wide range of consulting services, training and support.

QlikFramework

Also see Ted Cuzzillo blogpost about it here: http://datadoodle.com/2013/10/09/next-for-qlik/# and Cindi Howson’s old post here: http://biscorecard.typepad.com/biscorecard/2012/05/qliktech-shares-future-product-plans-for-qlikview.html and new article here: http://www.informationweek.com/software/business-intelligence/qliktech-aims-to-disrupt-bi-again/240162403#!

Today Tableau Customer Conference 2013 started with 3200+ attendees from 40+ countries and 100+ industries, with 700 employees of Tableau, 240 sessions. Tableau 8.1 pre-announced today for release in fall of 2013, also version 8.2 planned for winter 2014, and Tableau 9.0 for later in 2014.

Update 9/10/13: keynote now is available recorded and online:  http://www.tableausoftware.com/keynote
(Recorded Monday Sept 9, 2013 Christian Chabot, Chris Stolte and the developers LIVE)

New in 8.1: 64-bit, Integration with R, support for SAML, IPV6 and External Load Balancers, Copy/Paste Dashboards and worksheets between workbooks, new Calendar Control, own visual style, including customizing even filters, Tukey’s Box-and-Whisker Box-plot, prediction bands, ranking, visual analytics for everyone and everywhere (in the cloud now)

Planned and new for 8.2: Tableau for MAC, Story Points (new type of worksheet/dashboard with mini-slides as story-points), seamless access to data via data connection interface to visually build a data schema, including inner/left/right/outer visual joins, beautifying columns names, easier metadata etc, Web authoring enhancements (it may get into 8.1: moving quick filters, improvement for Tablets, color encoding.) etc.

8.1:  Francois Ajenstat announced: 64-bit finally (I asked for that for many years) for server processes and for Desktop, support for SAML (single-sign-ON on Server and Desktop), IPV6, External Load Balancers:

Francois

SAML8.1: Dave Lion announced R integration with Tableau:

DaveLion

r8.1: Mike Arvold announced “Visual Analytics for everyone”, including implemention of famous Tukey’s Box-and-Whisker Box-plot (Spotfire has it for a while, see it here: http://stn.spotfire.com/stn/UserDoc.aspx?UserDoc=spotfire_client_help%2fbox%2fbox_what_is_a_box_plot.htm&Article=%2fstn%2fConfigure%2fVisualizationTypes.aspx ),

better forecasting, prediction bands, ranking, better heatmaps:

MikeArvold8.1: Melinda Minch announced “fast, easy, beautiful”, most importantly copy/paste dashboards and worksheets between workbooks, customizing everything, including quick filters, new calendar control, own visual style, folders in Data Window etc…

MelindaMinch28.2: Jason King pre-announced the Seamless access to data via data connection interface to visually build a data schema, including inner/left/right/outer “visual” joins, beautifying columns names, default formats, new functions like DATEPARSE, appending data-set with new tables, beautifying columns names, easier metadata etc.

JasonKingSeamlessAccess2data28.2: Robert Kosara introduced Story Points (using new type of worksheet/dashboard with mini-slides as story-points) for new Storytelling functionality:

RobertKosara2

Here is an example of Story Points, done by Robert:

storypoints-4

8.2: Andrew Beers pre-announced Tableau 8.2 on MAC and he got a very warm reception from audience for that:

AndrewBeers3Chris Stolte proudly mentioned his 275-strong development team, pre-announced upcoming Tableau Releases 8.1 (this fall), 8.2 (winter 2014) and 9.0 (later in 2014) and introduced 7 “developers” who (see above Francois, Mike, Dave, Melinda, Jason, Robert and Andrew) discussed during this keynote new features (feature list is definitely longer and wider that recent “innovations” we saw from Qlikview 11.2 and even from Spotfire 5.5):

ChrisStolte2Christian Chabot opening keynote today… He said something important: current BI Platforms are not fast, nor easy, they are not beautiful and not for anyone and they are definitely not “anywhere” but only in designated places with appropriate IT personnel (compare with Tableau Public, Tableau Online, Tableau free Reader etc.) and it is only capable to produce a bunch of change requests from one Enterprise’s department to another, which will take long time to implement with any SDLC framework.

CEOChristian basically repeated what I am saying on this blog for many years, check it here https://apandre.wordpress.com/market/competitors/ : traditional BI software (from SAP, IBM, Oracle, Microstrategy and even Microsoft cannot compete with Tableau, Qlikview and Spotfire) is obsolete and dying and this is very direct challenge and threat to BI vendors (I am not sure if they understand that): your (BI that is) time is over and now it is time for Tableau (also for Qlikview and Spotfire but they are slightly behind now…).

Update on 11/21/13: Tableau 8.1 is available today, see it here: http://www.tableausoftware.com/new-features/8.1 and Tableau Public 8.1 is available as well, see it here: http://www.tableausoftware.com/public/blog/2013/11/tableau-public-81-launches-2226

While blog preserving my observations and thoughts, it preventing me to spend enough time to read what other people thinking and saying, so I created almost 2 years ago the extension of this blog in the form of 2 Google+ pages http://tinyurl.com/VisibleData and http://tinyurl.com/VisualizationWithTableau , where I accumulated all reading pointers for myself and gradually reading those materials when I have time.

Those 2 pages magically became extremely popular (this is unintended result) with total more than 5000 Google+ followers as of today. For example here is a Chart showing monthly growth of the  number of followers for the main extension of this blog http://tinyurl.com/VisibleData :

GPFollowersMonthly

So please see below some samples of Reading Pointers accumulated over last 3 months of summer by my Google+ pages:

Author trying to simplify BigData Definition as following: “BigData Simplified: Too much data to fit into a single server”: http://yottascale.com/entry/the-colorful-secrets-of-bigdata-platforms

Recent talk from Donald Farmer: http://www.wired.com/insights/2013/06/touch-the-next-frontier-of-business-intelligence/

Dmitry pointing to implementation Disaster of Direct Discovery in Qlikview 11.2: http://bi-review.blogspot.com/2013/04/first-look-at-qlikview-direct-discovery.html

Specs for Tableau in Cloud: https://www.tableausoftware.com/products/online/specs

The DB-Engines Monthly Ranking ranks database management systems according to their popularity. Turned out that only 3 DBMSes are popular: Oracle, SQL Server and MySQL:

According to Dr. Andrew Jennings, chief analytics officer at FICO and head of FICO Labs, three main skills of data scientist are the same 3 skills I tried to find when hiring programmers for my teams 5, 10, 20 and more years ago: 1. Problem-Solving Skills. 2. Communications Skills. 3. Open-Mindedness. This makes all my hires for last 20+ years Data Scientists, right? See it here: http://www.informationweek.com/big-data/news/big-data-analytics/3-key-skills-of-successful-data-scientis/240159803

A study finds the odds of rising to another income level are notably low in certain cities, like Atlanta and Charlotte, and much higher in New York and Boston: http://www.nytimes.com/2013/07/22/business/in-climbing-income-ladder-location-matters.html

Tableau is a prototyping tool: http://tableaufriction.blogspot.com/2013/07/the-once-and-future-prototyping-tool-of.html

Why More Data and Simple Algorithms Beat Complex Analytics Models: http://data-informed.com/why-more-data-and-simple-algorithms-beat-complex-analytics-models/

New Census Bureau Interactive Map Shows Languages Spoken in America: http://www.census.gov/newsroom/releases/archives/education/cb13-143.html

Google silently open sourced a tool called word2vec, prepackaged deep-learning software designed to understand the relationships between words with no human guidance. It actually similar to known for a decade methods called PLSI and PLSA:

“Money is not the only reward of education, yet it is surely the primary selling point used to market data science programs, and the primary motivator for students. But there’s no clear definition of data science and no clear understanding of what knowledge employers are willing to pay for, or how much they will pay, now or in the future. Already I know many competent, diligent data analysts who are unemployed or underemployed. So, I am highly skeptical that the students who will invest their time and money in data science programs will reap the rewards they have been led to expect.”: http://www.forbes.com/sites/gilpress/2013/08/19/data-science-whats-the-half-life-of-a-buzzword/

Some good blog-posts from InterWorks:

Technique for using Tableau data blending to create a dynamic, data-driven “parameter”: http://drawingwithnumbers.artisart.org/creating-a-dynamic-parameter-with-a-tableau-data-blend/

More about Colors:

Russian Postcodes are collected and partially visualized:

http://acuitybusiness.com/blog/bid/175066/Three-Reasons-Why-Companies-Should-Outlaw-Excel

EXASolution claims to be up to 1000 times faster than traditional databases and the fastest database in the world – based on in memory computing.
http://www.exasol.com/en/exasolution/technical-details.html

web interest to Tableau and Qlikview:
http://www.google.com/trends/explore?q=qlikview%2C+tableau%2C+spotfire%2C+microstrategy#q=tableau%2C%20microstrategy%2C%20qlikview%2C%20spotfire&geo=US&date=9%2F2008%2061m&cmpt=q

Tableau Software filed for IPO, on the New York Stock Exchange under the symbol “DATA”. In sharp contrast to other business-software makers that have gone public in the past year, Tableau is profitable, despite hiring huge number of new employees. For the years ended December 31, 2010, 2011 and 2012,  Tableau’s total revenue were $34.2 million, $62.4 million and $127.7 million for 2012. Number of full-time employees increased from 188 as of December 31, 2010 to 749 as of December 31, 2012.

Tableau’s biggest shareholder is venture capital firm New Enterprise Associates, with a 38 percent stake. Founder Pat Hanrahan owns 18 percent, while co-founders Christopher Stolte and Christian Chabot, who is also chief executive officer, each own more than 15 percent. Meritech Capital Partners controls 6.4 percent. Tableau recognized three categories of Primary Competitors:

  • large suppliers of traditional business intelligence products, like IBM, Microsoft, Oracle and SAP AG;

  • spreadsheet software providers, such as Microsoft Corporation

  • business analytics software companies: Qlik Technologies Inc. and TIBCO Spotfire.

TBvsQVvsSF

Update 4/29/13: This news maybe related to Tableau IPO: I understand that Microstrategy’s growth cannot be compared with growth of Tableau or even Qliktech. But to go below of the average “BI market” growth? Or even 6% or 24% decrease? What is going on (?) here : “First quarter 2013 revenues were $130.2 million versus $138.3 million for the first quarter of 2012, a 6% decrease.  Product licenses revenues for the first quarter of 2013 were $28.4 million versus $37.5 million for the first quarter of 2012, a 24% decrease.”

Update 5/6/13: Tableau Software Inc. will sell 5 million shares, while shareholders will sell 2.2 million shares, Tableau said in an amended filing with the U.S. Securities and Exchange Commission. The underwriters have the option to purchase up to an additional 1,080,000 shares. It means total can be 8+ millions of shares for sale.

The company expects its initial public offer to raise up to $215.3 million at a price of $23 to $26 per share. If this happened, that will create public company with large capitalization, so Qliktech and Spotfire will have an additional problem to worry about. This is how QLIK (blue line), TIBX (red) and MSTR (orange line) stock behaved during last 6 weeks after release of Tableau 8 and official Tableau IPO announcement:

QlikTibxMstr

Update 5/16/13: According to this article  at Seeking Alpha (also see S-1 Form) Tableau Software Inc. (symbol “DATA”) is scheduled a $176 million IPO with a market capitalization of $1.4 billion for Friday, May 17, 2013. Tableau’s March Quarter sales were up 60% from the March ’12 quarter. Qliktech’s sales were up only 23% on a similar comparative basis.

nyse

According to other article, Tableau raised it IPO price and it may reach capitalization of $2B by end of Friday, 5/17/13. That is almost comparable with capitalization of Qliktech…

Update 5/17/13: Tableau’s IPO offer price was $31 per share, but it started today

with price $47 and finished day with $50.75 (raising $400M in one day) with estimated Market Cap around $3B (or more?). It is hard to understand the market: Tableau Stock (symbol: DATA) finished its first day above $50 with Market Capitalization higher than QLIK, which today has Cap = $2.7B but Qliktech has almost 3 times more of sales then Tableau!

For comparison MSTR today has Cap = $1.08B and TIBX today has Cap = $3.59B. While I like Tableau, today proved that most investors are crazy, if you compare numbers in this simple table:

Symbol  : Market Cap, $B, as of 5/17/13 Revenue, $M, as of 3/31/13 (trailing 12 months) FTE (Full Time Employees)
TIBX 3.59 1040 3646
MSTR 1.08 586 3172
QLIK 2.67 406 1425
DATA between $2B and $3B? 143 834

See interview with Co-Founder of Tableau Software Christian Chabot  – he discusses taking the company’s IPO with Emily Chang on Bloomberg Television’s “Bloomberg West.” However it makes me sad when Tableau’s CEO is implying that Tableau is ready for big data, which is not true.

TableauCEOaboutIPOHere are some pictures of the Tableau team at the NYSE:  http://www.tableausoftware.com/ipo-photos and here is the announcement about “closing IPO”.

Initial public offering gave to Tableau $254 million (preliminary estimate)

Today Tableau 8 was released with 90+ new features (actually it may be more than 130) after exhausting 6+ months of Alpha and Beta Testing with 3900+ customers as Beta Testers! I personally expected it it 2 months ago, but I rather will have it with less bugs and this is why I have no problem with delay. During this “delay” Tableau Public achieved the phenomenal Milestone: 100 millions of users…

Tableau 8 introduced:

  • web and mobile authoring,
  • added access to new data sources: Google Analytics, Salesforce.com, Cloudera Impala, DataStax Enterprise, Hadapt, Hortonworks Hadoop Hive, SAP HANA, and Amazon Redshift.
  • New Data Extract API that allows programmers to load data from anywhere into Tableau and make certain parts of Tableau Licensing ridiculous, because consuming part of licensing (for example core licensing) for background tasks should be set free now.
  • New JavaScript API enables integration with business (and other web-) applications.
  • Local Rendering: leveraging the graphics hardware acceleration available on ordinary computers. Tableau 8 Server dynamically determines where rendering will complete faster – on the server or in the browser. Also – and acts accordingly. Also Dashboards now render views in parallel when possible.

Tableau Software plans to add in next versions (after 8.0) some very interesting and competitive features, like:

  • Direct query of large databases, quick and easy ETL and data integration.
  • Tableau on a Mac and Tableau as a pure Cloud offering.
  • Make statistical & analytical techniques accessible (I wonder if it means integration with R?).
  • Tableau founder Pat Hanrahan recently talked about “Showing is Not Explaining”, so Tableau planned to add features that support storytelling by constructing visual narratives and effective communication of ideas.

I did not see on Tableau’s roadmap some very long overdue features like 64-bit implementation (currently even all Tableau Server processes, except one, are 32-bit!), Server implementation on Linux (we do not want to pay Windows 2012 Server CAL taxes to Bill Gates) and direct mentioning of integration with R like Spotfire does – I how those planning and strategic mistakes will not impact upcoming IPO.

I personally think that Tableau has to stop using its ridiculous practice when 1 core license used per each 1 Backgrounder server process and since Tableau Data Extract API is free so all Tableau Backgrounder Processes should be free and have to be able to run on any hardware and even any OS.

Tableau 8 managed to get the negative feedback from famous Stephen Few, who questioned Tableau’s ability to stay on course. His unusually long blog-post “Tableau Veers from the Path” attracted enormous amount of comments from all kind of Tableau experts. I will be cynical here and notice that there is no such thing as negative publicity and more publicity is better for upcoming Tableau IPO.

TBvsQVvsSF

Today TIBCO announced Spotfire 5, which will be released in November 2012. Two biggest news are the access to SQL Server Analysis Services cubes and the integration with Teradata “by pushing all aggregations, filtering and complex calculations used for interactive visualization into the (Teradata) database”.

Spotfire team “rewrote” its in-memory engine for v. 5.0 to take advantage of high-capacity, multi-core servers. “Spotfire 5 is capable of handling in-memory data volumes orders of magnitude greater than the previous version of the Spotfire analytics platform” said Lars Bauerle, vice president of product strategy at TIBCO Spotfire.

Another addition is “in-database analysis” which allows to apply analytics within the database platforms (such as Oracle, Microsoft SQL Server and Teradata) without  extracting and moving data, while handling analyses on Spotfire server and returning result sets back to the database platform.

Spotfire added new Tibco Enterprise Runtime for R, which embeds R runtime engine into the Spotfire statistical server. TIBCO claims that Spotfire 5.0 scales to tens of thousands of users! Spotfire 5 is designed to leverage the full family of TIBCO business optimization and big data solutions, including TIBCO LogLogic®, TIBCO Silver Fabric, TIBCO Silver® Mobile, TIBCOBusinessEvents®, tibbr® and TIBCO ActiveSpaces®.

The short version of this post: as far as Data Visualization is a concern, the new Power View from Microsoft is the marketing disaster, the architectural mistake and the generous gift from Microsoft to Tableau, Qlikview, Spotfire and dozens of other vendors.

For the long version – keep reading.

Assume for a minute (OK, just for a second) that new Power View Data Visualization tool from Microsoft SQL Server 2012 is almost as good as Tableau Desktop 7. Now let’s compare installation, configuration and hardware involved:

Tableau:

  1. Hardware:  almost any modern Windows PC/notebook (at least dual-core, 4GB RAM).
  2. Installation: a) one 65MB setup file, b) minimum or no skills
  3. Configuration: 5 minutes – follow instructions on screen during installation.
  4. Price – $2K.

Power View:

  1. Hardware: you need at least 2 server-level PCs (each at least quad-core, 16GB RAM recommended). I will not recommend to use 1 production server to host both SQL Server and SharePoint; if you desperate, at least use VM(s).
  2. Installation: a) Each Server  needs Windows 2008 R2 SP1 – 3GB DVD; b) 1st Server needs SQL Server 2012 Enterprise or BI Edition – 4GB DVD; c) 2nd Server needs SharePoint 2010 Enterprise Edition – 1GB DVD; d) A lot of skills and experience
  3. Configurations: Hours or days plus a lot of reading, previous knowledge etc.
  4. Price: $20K or if only for development it is about $5K (Visual Studio with MSDN subscription) plus cost of skilled labor.

As you can see, Power View simply cannot compete on mass market with Tableau (and Qlikview and Spotfire) and time for our assumption in the beginning of this post is expired. Instead now is time to remind that Power View is 2 generations behind Tableau, Qlikview and Spotfire. And there is no Desktop version of Power View, it is only available as a web application through web browser.

Power View is a Silverlight application packaged by Microsoft as a SQL Server 2012 Reporting Services Add-in for Microsoft SharePoint Server 2010 Enterprise Edition. Power View is (ad-hoc) report designer providing for user an interactive data exploration, visualization, and presentation web experience. Microsoft stopped developing Silverlight in favor of HTML5, but Silverlight survived (another mistake) within SQL Server team.

Previous report designers (still available from Microsoft:  BIDS, Report Builder 1.0, Report Builder 3.0, Visual Studio Report Designer) are capable to produce only static reports, but Power View enables users to visually interact with data and drill-down all charts and Dashboard similar to Tableau and Qlikview.

Power View is a Data Visualization tool, integrated with Microsoft ecosystem. Here is a Demo of how the famous Hans Rosling Data Visualization can be reimplemented with Power View:

Compare with previous report builders from Microsoft, Power View allows many new features, like Multiple Views in a Single Report, Gallery preview of Chart Images, export to PowerPoint, Sorting within Charts by measures and Categories, Multiple Measures in Charts, Highlighting of selected data in reports and Charts, Synchronization of Slicers (Cross-Filtering), Measure Filters, Search in Filters (convenient for a long lists of categories), dragging data fields into Canvas (create table) or Charts (modify visualization), convert measures to categories (“Do Not Summarize”), and many other features.

As with any of 1st releases from Microsoft, you can find some bugs from Power View. For example, KPIs are not supported in Power View in SQL Server 2012, see it here: http://cathydumas.com/2012/04/03/using-or-not-using-tabular-kpis/

Power View is not the 1st attempt to be a full player in Data Visualization and BI Market. Previous attempts failed and can be counted as Strikes.

Strike 1: The ProClarity acquisition in 2006 failed, there have been no new releases since v. 6.3; remnants of ProClarity can be found embedded into SharePoint, but there is no Desktop Product anymore.

Strike 2: Performance Point Server was introduced in November, 2007, and discontinued two years later. Remnants of Performance Point can be found embedded into SharePoint as Performance Point Services.

Both failed attempts were focused on the growing Data Visualization and BI space, specifically at fast growing competitors such as Qliktech, Spotfire and Tableau. Their remnants in SharePoint functionally are very behind of Data Visualization leaders.

Path to Strike 3 started in 2010 with release of PowerPivot (very successful half-step, since it is just a backend for Visualization) and xVelocity (originally released under name VertiPaq). Power View is continuation of these efforts to add a front-end to Microsoft BI stack. I do not expect that Power View will gain as much popularity as Qlikview and Tableau and in my mind Microsoft will be a subject of 3rd strike in Data Visualization space.

One reason I described in very beginning of this post and the 2nd reason is absence of Power View on desktop. It is a mystery for me why Microsoft did not implement Power View as a new part of Office (like Visio, which is a great success) – as a new desktop application, or as a new Excel Add-In (like PowerPivot) or as a new functionality in PowerPivot or even as a new functionality in Excel itself, or as new version of their Report Builder. None of these options preventing to have a Web reincarnation of it and such reincarnation can be done as a part of (native SSRS) Reporting Services – why involve SharePoint (which is – and I said it many times on this blog – basically a virus)?

I am wondering what Donald Farmer thinking about Power View after being the part of Qliktech team for a while. From my point of view the Power View is a generous gift and true relief to Data Visualization Vendors, because they do not need to compete with Microsoft for a few more years or may be forever. Now IPO of Qliktech making even more sense for me and upcoming IPO of Tableau making much more sense for me too.

Yes, Power View means new business for consulting companies and Microsoft partners (because many client companies and their IT departments cannot handle it properly), Power View has a good functionality but it will be counted in history as a Strike 3.

In unusual, interesting (what it means? is it promising or what?) move the two Data Visualization leaders (Panopticon and Qliktech) partners today, see

http://panopticon.com/Panopticon-Software-Partners-with-QlikTech-to-Provide-Real-Time-Visual-Data-Monitoring-and-Analysis-Dashboards

“to offer enhanced, real-time visualization capabilities for the QlikView Business Discovery platform”.

Panopticon’s press-release looks overly submissive to me:

“As a member of QlikTech’s Qonnect Partner Program for Technology Partners, Panopticon supports QlikView desktop, web, and mobile interactive dashboards and allows users to filter and interact directly with real-time data. By integrating Panopticon into their systems, QlikView users can:

The combined Panopticon-QlikView platform is now available for immediate installation.”

Panopticon integration into QlikView dashboards utilizes QlikView UI extension objects within the web browser. The extension object calls Panopticon “web parts” and creates a Panopticon extension object with a number of pre-defined properties. The defined context/data is passed into the Panopticon extension object. The Panopticon “web parts” call a Panopticon EX Java applet and renders the requested Panopticon visualization workbook within the context defined by the QlikView user. The Panopticon component executes parameterized URL calls and parameterized JavaScripts to update the parent QlikView display.

Qliktech is trying to be politically correct and its Michael Saliter, Senior Director Global Market Development – Financial Services at QlikTech said, “Our partnership with Panopticon allows us to incorporate leading real-time visualization capabilities into our QlikView implementations. We recognize the importance of providing our clients with truly up-to-date information, and this new approach supports that initiative. Our teams share a common philosophy about proper data visualization design. This made it easy to develop a unified approach to the presentation of real-time, time series, and static data in ways that people can understand in seconds.”

While I like when competitors are cooperating (it benefits users and hopefully improve sales for both vendors), I still have a question: Qliktech got a lot of money from IPO, had a lot of sales and hired a lot of people lately; why they (Qlikview Developers) was not able to develop real-time functionality themselves?

Hugh Heinsohn, VP of Panopticon, said to me: “we (Panopticon) don’t see ourselves as competitors – and neither do they (Qliktech). When you get into the details, we do different things and we’re working together closely now”

Another indirect sign of relationship between Panopticon and Qliktech is the recent inclusion of Måns Hultman, former CEO of QlikTech into the list of advisors for Panopticon’s Board of Directors.

Other questions are rising too: if Qliktech suddenly is open to integration with Panopticon, why not to integrate with Quantrix and R Library (I proposed integration with R a while ago). Similar questions applicable to Tableau Software…

Dan Primack, Senior Editor at Fortune, posted today at http://finance.fortune.cnn.com/2012/02/22/tableau-to-ipo-next-year/ a suggestion that Tableau can go public next year and I quote:

“Scott Sandell, a partner with New Enterprise Associates  (the venture capital firm that is Tableau’s largest outside shareholder) told Dan “that the “board-level discussions” are about taking the company public next year, even though it has the numbers to go out now if it so chose. Sandell added that the company has been very efficient with $15 million or so it has raised in VC funding, and that it shouldn’t need additional pre-IPO financing”.

Mr. Primack also mentioned an “unsolicited email, from outside spokesman: “Next week Tableau Software will announce its plans to go IPO“…

I do not have comments, but I will not be surprised if somebody will buy Tableau before IPO… Among potential buyers I can imagine:

  • Microsoft (Seattle, Multidimensional Cubes, integration with Excel),
  • Teradata (Aster Data is in, front-end for “big data” is needed),
  • IBM (if you cannot win against the innovator, how about buying it),
  • and even Oracle (everything moving is the target?)…

New version 3.3 of SpreadsheetWEB with new features like Data Consolidation, User Groups, Advance Analytics and Interactive Charts, is released this month by Cambridge, MA-based Pagos, Inc.

SpreadsheetWEB is known as the best SaaS platform with unique ability to convert Excel spreadsheets to rich web applications with live database connections, integration with SQL Server, support for 336 Excel functions (see full list here http://wiki.pagos.com/display/spreadsheetweb/Supported+Excel+Formulas ), multiple worksheets, Microsoft Drawing, integration with websites and the best Data Collection functionality among BI tools and platforms.

SpreadsheetWEB supports Scripting (Javascript), has own HTML editor, has rich Data Visualization and Dashboarding functionality (32 interactive Chart types are supported, see http://spreadsheetweb.com/support_charts.htm ),

See the simple Video Tutorial about how to create a Web Dashboard with Interactive Charts by publishing your Excel Spreadsheet using SpreadsheetWEB 3.3 here:

SpreadsheetWEB supports Mapping for a while, see video showing how you can create Map application in less then 4 minutes:

as well as PivotTables, Web Services, Batch Processing, and many other new features, see it here: http://spreadsheetweb.com/features.htm

In order to create a SpreadsheetWEB application, all you need is Excel and free SpreadsheetWEB Add-in for Excel, see many impressive online Demos here: http://spreadsheetweb.com/demo.htm

I said on this blog many times that 80% of Data Visualization (DV) is … Data.

SQL Server 2012 is here.

And technology and process of how these Data collected, extracted, transformed and loaded into DV backend and frontend is a key to DV success. It seems to me that one of the best possible technology for building DV backend is around the corner as SQL Server 2012 will be released soon – Release Candidate for it is out…

And famous Microsoft marketing machine is not silent about it. SQL Server 2012 Virtual Launch Event planned for March 7, 2012 and real release probably at the end of March 2012.

Columnstore Index.

I already mentioned on this blog the most interesting feature for me – the introduction of Columnstore Index (CSI) can transform SQL Server into Columnar Database (for DV purposes) and accelerates DV-relevant Queries by 10X or even 100X of times. Oracle does not have it!

.

Some reasonable rules and features applied to CSI: each table can have only one CSI; CSI has Row grouping (about million rows, like paging for columns); table with CSI cannot be replicated. New (unified for small and large memory allocations) memory manager optimized for Columnstore Indexes, supports Windows 8 maximum memory and logical processors.

Power View.

SSRS (Reporting Services) got massive improvements, including new Power View as Builder/Viewer of interactive Reports. I like this feature: “even if a table in the view is based on an underlying table that contains millions of rows, Power View only fetches data for the rows that are visible in the view at any one time” and UI features (some of them are standard for existing Data Visualization tools, like multiple views in Power View reports (see gallery of thumbnails in the bottom of screenshot below):

.

“2 clicks to results”, export to PowerPoint etc. See also video here:

.

.

PowerView is still far behind Tableau and Qlikview as a Visualizer, but at least it makes SSRS reports more interactive and development of them easier. Below are some thumbnails of Data Visualization samples produced with PowerView and presented by Microsoft:

Support for Big Data.

SQL Server 2012 has a lot new features like “deep” HADOOP support (including Hive ODBC Driver) for “big data” projects, ODBC drivers for Linux, grouping databases into Availability Group for simultaneous failover, Contained Databases (enable easy migration from one SQL Server instance to another) with contained Database users.

Parallel Data Warehouse, Azure, Data Explorer.

And don’t forget PDW (SQL Server-based Parallel Data Warehouse;  massive parallel processing (MPP) provides scalability and query performance by running independent servers in parallel with up to 480 cores) and SQL Azure cloud services with it high availability features…

.

New Data Explorer allows discover data in the cloud and import them from standard and new data sources, like OData, Azure Marketplace, HTML etc. and visualize and publish your Data to the cloud.

LocalDB.

LocalDB is a new free lightweight deployment option for SQL Server 2012 Express Edition with fewer prerequisites that installs quickly. It is an embedded SQL Server database for desktop applications (especially for DIY DV apps) or tools. LocalDB has all of the same programability features as SQL Server 2012 Express, but runs in user mode with applications and not as a service. Application that use LocalDB simply open a file. Once a file is opened, you get SQL Server functionality when working with that file, including things like ACID transaction support. It’s not intended for multi-user scenarios or to be used as a server. (If you need that, you should install SQL Server Express.)

BIDS.

SQL Server 2012 is restoring a very desirable feature, which was missing in Visual Studio 2010 for 2+ years – something called BIDS (BI Development Studio was available as part of Visual Studio 2008 and SQL Server 2008). For that a developer needs VS2010 installed with SP1 and then install “SQL Server Data Tools” (currently it is in the state of CTP4, but I guess it will be a real thing when when SQL Server 2012 will be released to production).

SSAS, Tabular Mode, PowerPivot, DAX.

Most important improvement for BI and Data Analytics will be of course the changes in SSAS (SQL Server Analysis Services), including the addition of  Tabular Mode, restoration of BIDS (see above), the ability to design local multidimensional cubes with PowerPivot and Excel and then deploy them directly from Excel as SSAS Cubes, the new DAX language shared between PowerPivot and SSAS, and availability of all those Excel Services directly from SSAS without any need for SharePoint. I think those DV tools who will able to connect to those SSAS and PowerPivot Cubes will have a huge advantage. So far only Tableau has it (and Omniscope has it partially).

Backend for Data Visualization.

All of these features making SQL Server 2012 a leading BI stack and backend for Data Visualization applications and tools. I just wish that Microsoft will develop an own DV front-end tool, similar to Tableau or Qlikview and integrate it with Office 201X (like they did with Visio), but I guess that DV market ( approaching $1B in 2012) is too small compare with markets for Microsoft Office and SQL Server.

Pricing.

Now is time for a “bad news”. The SQL Server 2012 CAL price will increase by about 27%. New pricing you can see below and I predict you will not like it:

I never liked pre-announcements of “new” products, especially if they are in state which will screw my PCs. But almost everybody doing it to us, starting with Microsoft SQL Server 2012 (Denali can be downloaded as “CTP3”), Tableau 7.0, Qlikview 11 (Qliktech partners and customers can download “Release candidate”) etc. Just a few months after releasing Spotfire 3.3, TIBCO announced that Spotfire 4.0 will be available in November 2011 with a lot of new features.

.

Some of them sound like buzzwords: “”free dimensional” analytics, collective intelligence, visual and social data discovery etc.” (we need that marketing will brainwash us, right?), but some of them can be very useful, like integration with TIBBR (that I like; in fact TIBCO has many other good products and they should be integrated with Spotfire) and SharePoint (sounds like a M$ bending to me, I don’t see too much DV money coming from SharePoint hole), support for dynamic icons, sparklines,

stepped linecharts, pop-over filters and legends, better font management, embedded actions and more. Some features I wish will be added, but I guess we need to wait more: I wish to be able to read with Spotfire the SSAS and PowerPivot multidimensional Cubes and support for some other Data Sources, like Tableau 6.1 does…

.

Spotfire and its Web Player Server support  now the latest web browsers, .NET 4.0 and it dropped support for obsolete stuff like Internet Explorer 6 and Windows 2003 Server. I mentioned on this blog earlier that I like Spotfire Silver 2.0 and the wealth and depth of Spotfire Analytical Platform (S-Plus, Miner, S+FinMetrics, Spotfire Developer/API, Statistics, Data and Automation Services, Metrics, Network Analysis, Decision Site, Clinical Graphics and more, this list should make Qliktech and Tableau worry or at least try to add similar features…).

.

Spotfire updated their set of Demos to reflect Spotfire 4.0 features: Spotfire Demos and Templates. More to come later, especially when Spotfire 4.0 will be Released (as oppose to be announced).

Spreadsheets (VisiCalc or “Visible Calculator” was released by Dan Bricklin and Bob Frankston in October 1979 – 32 years ago – originally for Apple II computer) were one of the very first Business Intelligence (BI) software (sold over 700,000 copies in six years).

It was released on October 19, 1979 – see the original Diary of Dan about it (also see notes of Peter Jennings here and here and especially detailed Bob Frankston’s article here):

For historical purposes I have to mention that VisiCalc actually was not the first spreadsheet program invented (for example I am aware of multi-user spreadsheet software written before VisiCalc in USSR in PL/1 for mainframes with IBM’s IMS Database as a backend ), but it is a first commercial spreadsheet introduced on American market and it was a turning point of PC industry.

The “Visible Calculator” went on sale in November of 1979 and was a big hit. It retailed for US$100 and sold so well that many dealers started bundling the Apple II with VisiCalc. The success of VisiCalc turned Apple into a successful company, selling tens of thousands of the pricey 32 KB Apple IIs (no matter how hard Bob Frankston tried, he could not fit VisiCalc in the 16 KB of RAM on the low-end Apple II. VisiCalc would only be available for the much more expensive 32 KB Apple II) to businesses that wanted them only for the spreadsheet. Version of VisiCalc for Atari was even retailed for $200!

VisiCalc was published without any Patent and it is a living prove that Patent System currently is useless for people, abused by large corporations for their own benefit, and it is actually a brake for innovations and it is not protecting inventors. Absence of patent protection for VisiCalc created the Spreadsheet Revolution and Innovations (SuperCalc, Lotus 1-2-3, QuattroPro, Excel, OpenOffice’s Calc, Google’s Spreadsheets and many others) and tremendously accelerated PC industry.

As Dan Bricklin said it by himself “We all borrowed from each other” and as George Bernard Shaw said: If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.”

Application of Spreadsheets in the BI field began with the integration of OLAP (On-Line Analytical Processing) and Pivot tables. In 1991, Lotus (in addition to 1-2-3) released Improv with Pivoting functionality (also see Quantrix as a reborned [originally in 1994-95] Improv), followed by Microsoft’s release (in Excel 5) of PivotTable in 1993 (trademarked by Microsoft). 500+ millions people currently using Excel and at least 5% of them using it for BI and Data Visualization purposes. PowerPivot added to Excel 2010 speedy and powerful in-memory columnar database which enables millions of end-users to have a self-serviced BI.

Essbase was the first scalable OLAP software to handle large data sets that the early spreadsheet software was incapable of. This is where its name comes from: Extended Spread Sheet Database (Essbase owned by Oracle now).  Currently one of the best OLAP and BI software is SSAS (Analysis Services from Microsoft SQL Server 2008 R2 and upcoming SQL Server 2012 with its new Tabular mode) and Excel 2010 with its PowerPivot, PivotTables and Pivot Charts is one of the most popular front-end for SSAS.

There is no doubt that Excel is the most commonly used software for “BI purposes”. While Excel is general business software, its flexibility and ease of use makes it popular for data analysis with millions of users worldwide. Excel has an install base of hundreds of millions of desktops: far more than any other “BI platform”. It has become a household name.With certain precaution it can be used for a good or at least prototyping Data Visualization (most of charts below created with Excel):

From educational utilization to domestic applications to prototyping (or approximated) Data Visualization and enterprise implementation, Excel has been proven incredibly indispensable. Most people with commercial or corporate backgrounds have developed a proficient Excel skillset. This makes Excel the ultimate self-service BI platform and spreadsheet technologies as a common ground for all viable Data Visualization technologies on market.

Oracle’s timing for “unveiling Exalytics In-Memory Machine” was unfortunate because it was in a shadow of Steve Jobs. In addition It was a lot of distraction between Larry Ellison’s and Mark Benioff’s egos.

Oracle is late to Analytics appliance game and have to fight already released products like Netezza/IBM (proven performer), SAP HANA (has large sales pipeline already), family of Teradata Appliances (Teradata Columnar coming in 2 months and sounds very good to me plus it packaged with Information Builders BI) , EMC/Greenplum Data Computing Appliance (doubled the sales during last year!), Microsoft Parallel Data Warehouse Appliance (Based on CTP3 I expect the great things from SQL Server 2011/2012/Denali) etc. They all are in-memory Machine, capable to store and process big data (exabytes? I guess depends on price…), almost all of them already have or will have soon columnar database.

Larry Ellison claimed during Oracle Openworld this week that “Exalytics is 10x faster than…just about everything.”

Yes, It runs a software stack that includes parallelized versions of Oracle’s TimesTen in-memory database and memory-optimized Essbase OLAP Server (“BI Foundation”), but it is not a columnar database, so I wonder how Oracle is going to prove Larry’s bold claims. However, Oracle TimesTen In-Memory Database for Exalytics supports columnar compression that reduces the memory footprint for in-memory data. Compression ratios of 5X are practical and help expand in-memory capacity (Qlikview, PowerPivot and Spotfire can do much better “columnar compression” then 5 times, claimed by Oracle)

Hardware itself looks impressive with four Intel Xeon© E7-4800 series processors (40 cores total) and 1TB of RAM but pricing is unclear. It has total 8 high speed ports:

  • 2 quad-data rate (QDR) 40 GB/s InfiniBand ports. When connected to Oracle Exadata, Oracle Exalytics becomes an integral part of the Oracle Exadata private InfiniBand network and has high-speed, low latency access to the database servers. When multiple Oracle Exalytics machines are clustered together, the InfiniBand fabric also serves as the high-speed cluster interconnect.
  • Exalytics has Two 10 GB/s Ethernet ports for connecting to enterprise data sources
  • Exalytics has Four 1 GB/s Ethernet ports are available for client access

Exalytics includes 3.6TBs of raw disk capacity. Optionally, clusters of Oracle Exalytics machines can leverage network attached storage.

Hardware portion of it probably below $100000 (I saw a guesstimate of $87000) but most expensive probably will be the Essbase (Business Intelligence Foundation Suite with in-memory Cubes now and ability to replicate entire data warehouse into TimesTen in-memory database) with list price about $450000, so we are talking  here about millions of dollars, which is (let’s wait and see the final pricing) will definitely reduce the number of potential buyers, especially considering weak Data Visualization and average BI functionality of Oracle’s software stack. According to Larry Ellison, Exalytics has 1TB of RAM but can hold five to 10TB of data in memory thanks to COLUMNAR compression.

Oracle Exalytics promotes self service analytics and makes it easier to develop analytics content by introducing a Presentation Suggestion Engine (PSE) which provides recommendations on type of visualizations to use to best represent a data set.

I do not expect anything spectacular from this “PSE”. For example Oracle proudly introduced “new micro charts and multi-panel trellis charts to visualize dense multi-dimensional, multi-page data on a single screen. The multi-panel trellis charts are particularly effective at displaying multiple visualizations across a common axis scale for easy comparison, to see a trend and quickly gain insights”:

but this micro charts available in much better shape and form for many years from Spotfire, Qlikview, Tableau etc. and relatively recently even from Excel.

In any case, Exalytics suppose to be well integrated with Oracle’s Exadata database machine and Exalogic application server. Mr. Ellison did some other bold claims like:

  • “For a given task, it will cost you less on an Exadata than it would on a plain old commodity server.”
  • “we move data around a hundred times faster than anyone else in this business”
  • “1,000 Exadata machines have been installed and 3,000 more will be sold this year”
  • “Java applications’ response times are 10 times as fast on Exalogic, and companies can serve many more users at once”

Special Note about Java.

I am not sure why Java is advantage for Oracle. Java is not welcome at Apple (can you say Objective C?), at Microsoft (can you cay C# ?) and recently even at Google (after Oracle sued Google for “misuse” of Java, which reminded me the Sun, disappearing after it sued Microsoft for … “misuse” of  … Java). Together those 3 companies have almost all cash (almost $200B if you exclude Oracle as a Java Owner) software companies have worldwide (Apple has $76B+ in a bank, Microsoft has $60B+ and Google has about $40B – may be less after buying Motorola Mobility) and I am simply following the money here. If Oracle wishes to have the Java-based advanced Data Visualization, they are better buy Visokio and integrate their  Omniscope with Exalytics and Exalogic instead of the inventing the wheel with PSE.

Teradata sounds good and smells like money, especially today. I already mentioned that they received U.S. Patent #7966340 on June 21, 2011. The patent is about SQL-MapReduce technology: the data analytic framework that combines the popular MapReduce™ software with the enterprise friendliness of SQL. (Also see article about “multi-structured data sources” from Aster  Data).

Today Teradata Columnar is announced (available in December 2011 as a component of Teradata Database 14) and Teradata Database 14 is released. The new columnar capability from Teradata allows users to mix-and-match (“hybrid”) columnar and row-based physical storage when it best suits an application. Teradata Columnar is integrated with the row-based storage and relational database software. Only the data in the columns required for a query are pulled into memory for processing, reducing the time-constraining input/output of a row-based approach that would read data from all the columns.

Teradata Columnar brings traditional “columnar” benefit: the flexible data compression. Teradata Columnar dynamically adjusts the compression mechanisms for optimal storage depends on type and size of data involved, automatically chooses from among six types of compression: run length, dictionary, trim, delta on mean, null and UTF8 based on the column demographics.

Again, these are just a good sound bites until Teradata Columnar will be released. Teradata may be trying to out-market Microsoft with its SQL Server 2011 (or Denali; as of today available as CTP3 community release) which already has the Columnstore Index, integrated with row-based storage and relational database.

I am wondering if Tableau will able timely and natively support Teradata Columnar as it supports now the Teradata Database (important for Data Visualization applications):

Today Tableau 6.1 is released (and client for iPad and Tableau Public for iPad), that includes the full support for incremental Data updates whether they are scheduled or on demand:

New in Tableau 6.1

  • Incremental Data updates scheduled or on demand
  • Text parser faster, can parse any text files as data source (no 4GB limit)
  • Files larger than 2GB can now be published to Tableau Server (more “big data” support)
  • Impersonation for SQL Server and Teradata; 4 times faster Teradata reading
  • Tableau Server auto-enables touch, pinch, zoom, gesture UI for Data Views
  • Tableau iPad app is released, it browses and filters a content on Server
  • Any Tableau Client sees Server-Published View: web browser, mobile Safari, iPad
  • Server enforces the same (data and user) security on desktop, browser, iPad
  • Straight links from an image on a dashboard, Control of Legend Layout etc.

Here is a Quick demo of how to create Data Visualization with Tableau 6.1 Desktop, how easy to publish it on Tableau server 6.1 and how it is instantly visible, accessible  and touch optimized on the iPad:

 

New since Tableau 6.0, more then 60 features, including:

  • Tableau now has in-memory Data Engine, which greatly improves I/O speed
  • Support for “big” data
  • Data blending from multiple sources
  • Unique support for local PowerPivot Multidimensional Cubes as Data Source
  • Support for Azure Datamarket and OData (Open Data Protocol) as Data Sources
  • Support for parameters in Calculations
  • Motion Charts and Traces (Mark History)
  • In average 8 times faster of rendering of Data Views (compare with previous version)

Tableau Product Family

  • Desktop: Personal ($999), Professional ($1999), Digital, Public.
  • Server: Standard, Core Edition, Digital, Public Edition.
  • Free Client: Web Browser, Desktop/Offline Tableau Reader.
  • Free Tableau Reader enables Server-less distribution of Visualizations!
  • Free Tableau Public served 20+ millions visitors since inception

Tableau Server

  • Easy to install: 13 minutes + optional 10 minutes for firewall configuration
  • Tableau has useful command line tools for administration and remote management
  • Scalability: Tableau Server can run (while load balancing) on multiple machines
  • Straightforward licensing for Standard Server (min 10 users, $1000/user)
  • With Core Edition Server License: unlimited number of users, no need for User Login
  • Digital Server Licensing based on impressions/month, allows unlimited data, Tableau-hosted.
  • Public Server License: Free, limited (100000 rows from flat files) data, hosted by Tableau.

Widest (and Tableau optimized) Native Support for data sources

  • Microsoft SSAS and PowerPivot: Excel Add-in for PowerPivot, native SSAS support
  • Native support for Microsoft SQL Server, Access, Excel, Azure Marketplace DataMarket
  • Other Enterprise DBMSes: Oracle, IBM DB2, Oracle Essbase
  • Analytical DBMSes: Vertica, Sybase IQ, ParAccel, Teradata, Aster Data nCluster
  • Database appliances: EMC/GreenPlum, IBM/Netezza
  • Many Popular Data Sources: MySQL, PostgreSQL, Firebird, ODBC, OData, Text files etc.

Some old problems I still have with Tableau

  • No MDI support in Dashboards, all charts share the same window and paint area
  • Wrong User Interface (compare with Qlikview UI) for Drilldown Functionality
  • Tableau’s approach to Partners is from stone ages
  • Tableau is 2 generations behind Spotfire in terms of API, Modeling and Analytics

Microsoft finally released SQL Server 11 “Denali” as CTP3 (Community Technology Preview) for public … Preview. Microsoft is (these are politeness words I can type) stubbornly refusing to have/build own Data Visualization Product. I doubt Crescent “experience” can be considered as a product, especially because it is Silverlight-base, while world already moved to HTML5.

If you have 7 minutes, you can watch Crescent Demo from WPC11, which is showing that while trailing a few years behind DV Leaders and Google, Microsoft is giving to its die hard followers something to cheer about:

I have to admit, that while there is nothing new (for DV expert) in video above, it is a huge progress compare with Excel-based Data Visualizations, which Microsoft tried to promote as a replacement of ProClarity and PerformancePoint Server. Even Microsoft itself positions Crescent (which is 32-bit only!) as a replacement for SSRS Report Builder, so DV Leaders can sleep well another night.

However, Microsoft’s BI Stack is the number 4 or 5 on my list of DV Leaders and CTP3 is so rich with new cool functionality, that it deserves to be covered on this blog.

Of course major news is availability of Tabular Data Model, which means VertiPaq in-memory columnar Engine, similar to PowerPivot Engine but running on Server without any SharePoint (which is a slow virus, as far as I am concerned) and without stupid SharePoint UI and limitations and I quote Microsoft: ” In contrast with the previous release, where VertiPaq was only available via in PowerPivot for SharePoint, you can now use VertiPaq on a standalone Analysis Services instance with no dependency on SharePoint.“!

SSAS (SQL Server Analysis Services) has new (they may existed before, but before CTP3 – ALL who knew that were under NDA) features like memory paging (allows models to be larger than the physical memory of the server, means unlimited scalability and BIG Data support), row level security (user identity used to hide/show visible data), KPI, Partitions; CTP3 removes the maximum 4GB file size limit for string storage file, removes the limit of 2 billion rows per table (each column is still limited to a maximum of 2 billion distinct values, but in columnar database it is much more tolerable restriction!).

New version of PowerPivot is released with support of  Tabular Model and I quote: “You can use this version of the add-in to author and publish PowerPivot workbooks from Excel 2010 to Microsoft SQL Server” and it means no SharePoint involvement again! As Marco Russo put it: “Import your existing PowerPivot workbooks in a Tabular project (yes, you can!)” and I agreed 100% with Marco when he said 4 times: Learn DAX!

After 3 years of delays, Microsoft is finally has BIDS for Visual Studio 2010  and that is huge too, I quote again: “The Tabular Model Designer … is now integrated with Microsoft SQL Server “Denali” (CTP 3) Business Intelligence Development Studio.” It means that BIDS now is not just available but is the main unified development interface for both Multidimensional and Tabular Data Models. Now we can forget about Visual Studio 2008 and finally use more modern VS2010!

Another extremely important for Data Visualization feature is not in SSAS but in SQL Server itself: Columnstore index is finally released and I a quote 1 more time again: “The … SQL Server (CTP 3) introduces a new data warehouse query acceleration feature based on a new type of index called the columnstore. This new index … improves DW query performance by hundreds to thousands of times in some cases, and can routinely give a tenfold speedup for a broad range of decision support queries… columnstore indexes limit or eliminate the need to rely on pre-built aggregates, including user-defined summary tables, and indexed (materialized) views. Furthermore, columnstore indexes can greatly improve ROLAP performance” (ROLAP can be used for real-time Cubes and real-time Data Visualizations).

All these cool SQL Server 11 new stuff is coming soon into Azure Cloud and this can be scary for any DV vendor, unless it knows (Tableau does; Qliktech and Spotfire still ignore SSAS) how to be friendly with Microsoft.

As we know now the newly coined by Microsoft term BISM (Business  Intelligence  Semantic Model) was a marketing attempt to have a “unified” umbrella

for 2 different Data Models and Data Engines: Multidimensional Cubes (invented by Mosha Pasumansky 15 years ago and the foundation for SSAS and MDX – SQL Server Analysis Services) and Tabular Model (used in PowerPivot and VertiPaq in-memory columnar Database with new DAX Language which is going to be very important for future Data Visualization projects).

New CTP3-released BIDS 2010 (finally almighty Visual Studio 2010 will have a “Business Intelligence Development Studio” after 3+ years of unjustified delays!) UI-wise will able to handle these 2 Data Models, but it is giving me a clue why Mosha left Microsoft for Google. And lack of DV product is a clue for me why Donald Farmer (face of Microsoft BI) left Microsoft for Qliktech.

Even more: if you need both Data Models to be present, you need to install 2 (TWO!) different instances of “Analysis Services”: one with Multidimensional Engine and one with new Tabular (VertiPaq/PowerPivot) Engine. It seems to me not as ONE “BI” architecture but TWO “BI” Architectures, interface-glued on Surface by BIDS 2010 and on back-end by all kind of Data Connectors. Basically Microsoft is in confused BI state now because financially it can afford 2 BI Architectures and NO Data Visualization Product!

I cannot believe I am saying this, but I wish Bill Gates back from retirement (it will be good for Microsoft shares and good for Microsoft market capitalization too – just ask Apple’s shareholders about Steve and they will say he is a god)!

Permalink: https://apandre.wordpress.com/2011/07/14/tabular-model/

In last few days something (3 news covered here in one post below) important for the future of Data Visualization and Big Data Analytics happened. IBM recently had 100th Birthday and almost at the same time their engineers published new invention, based on PCM (Phase-Change Memory).

  • PCM will not lose data when when power is turned off.
  • PCM 100 times faster (10 microseconds latency!) then flash and HDD.
  • PCM can endure at least 10 million write cycles (Flash maxed-out @30000)
  • PCM is cheap, has huge capacity and will be mass-produced before 2016.
  • PCM can be used everywhere from huge servers to smartphones


This invention is changing the approach to how to store and access “Big Data” and what portion of “Big Data” need to be in-memory (RAM) for Data Visualization purposes as oppose to outside of RAM (say on hard disk, flash or PCM). IBM may have a keys to Big Data kingdom…

To some people it may be unrelated, but not to me: Teradata just got the Patent on SQL-MapReduce technology they got from Aster Data acquisition. This technology allows also to integrate with Apache Hadoop and derived database systems, used in many Big Data applications.

And last but not least is a recent acknowledgment (for some reason it came from India’s branch of IBM Software and I am wondering why, but finally it came “Straight from the horse’s mouth”! ) from IBM that Data Visualization is the future of Business Intelligence (I said THIS many years ago and still repeating it from time to time: DV is new BI or in other words: the BI is dead, all hails to DV!). IBM is very proudly saying that Cognos 10 supports “enormous” number of Charts (I guess it will make Qlikview, Spotfire and Tableau people laughing)

and that the most discussed feature in Cognos 10 is Active Reports. This functionality allows the report authors to create interactive reports (apparently it is a big deal for IBM!).

IBM even is spreading rumors for weeks (through people who signed NDA with them) about Cognos TM1-based “new visualization tool”, which will “disrupt” DV market… I guess because IBM knows that BI is dead (and IBM wasted $14+B buying 24 BI companies lately) and DV is new BI.

Since IBM improved PCM (see above) and had 100th birthday, I really wish good luck to them, but I wish IBM to stay focused on what they good at instead of spreading all over the high-tech. All these 3 “news” were published yesterday and today and somehow connected in my mind to Data Visualization’s future and forced me to publish this “eclectic” post…

TIBCO released Spotfire 3.3 and first (see what is new here) that jumped to my eyes was how mature this product is. For example, among new features is improved scalability – each additional simultaneous user of a web analysis initially claims very little additional system memory:

Many Spotfire customers will be able to support a greater number of web users on their existing hardware by upgrading to 3.3. Spotfire Web Player 3.3 includes significant improvements in memory consumption (as shown above for certain scenarios). Theoretically goal is to minimize the amount of system memory needed to support larger numbers of simultaneous users on the same analysis file. Main use case here: the larger the file and the greater the number of simultaneous web users on that file, then less initial system memory required to support each additional user: it is greatly reduced compared to version 3.2.1 and earlier.

Comparison with competition and thorough testing of new Spotfire scalability has to be done (similar to what Qliktech done with Qlikview here), but my initial reaction is as I said in a Title: we are witnessing a very mature software. Apparently the Defense Intelligence Agency (DIA) agrees with me and Defense Intelligence Agency Selects TIBCO Spotfire Analytics Solutions for Department of Defense Intelligence Information System Community. “With more than 16,500 military and civilian employees worldwide, DIA is a major producer and manager of foreign military intelligence”

Spotfire 3.3 also includes collaborative bookmarking, which enables all Spotfire users  to capture a dashboard – its complete configuration, including markings, drop down selections, and filter settings and share that visualization immediately with other users of that same dashboard, regardless of client in use. Spotfire actually not just a piece of Data Visualization Software, but a real Analytical Platform with large portfolio of products, including completely integrated S-PLUS (commercial version of R Library which has million of users), best Web Client (you can go Zero-footprint with Spotfire Web Player or/and partially free Spotfire Silver), free iPad Client version 1.1.1 (requires iTunes, so be prepared for Apple intrusion), very rich API, SDK, integration with Visual Studio, support of IronPython and JavaScript , well-thought Web Architecture, set of Extension Points etc.

System requirements for Spotfire 3.3 can be found here. Coincidentally with 3.3 Release Spotfire VAR Program got expansion too. Spotfire has a very rich set of training options, see it here. You can also find set of good Spotfire videos from Colin White’s Screencast Library, especially 2011 Webcasts.

My only and large concern with Spotfire is its focus, since it is part of a large corporation TIBCO, which has 50+ products and 50+ reasons to focus on something else. Indirectly it can be confirmed with sales: my estimate that Tableau is growing much faster than Spotfire (sales-wise) and Qlikview Sales probably 3 times larger (dollar-wise) than Spotfire sales. Since TIBCO bought Spotfire in 2007, I expected Spotfire will be integrated with other great TIBCO products, but after 4 years it is still not a case… And TIBCO has no reason to change its corporate policies, since its busines is good and stock is doing well:

(at least 500% increase of share price since end of 2008!). Also see article written by Ted Stamas for SeekingAlpha and comparison of TIBX vs. ETF here:

I think it is interesting to notice that TIBCO recently rejected a buyout offer from HP!

Last week of April 2011 was good for Qliktech. It released the results for a First Quarter 2011 and they are very positive.

Revenue is up (does not look like it is slowing down) 44% YoY, if compared with 1Q2010 with revenue $63M and projection for total 2011 now about $300M (up from preliminary projection of $280M before Q1 happened). Ended the first quarter of 2011 with an active customer count of approximately 19,000 (means about 900000 licensed, paying Data Visualization and BI users now and number of Qlikview users may exceed 1 million in 2011!), up from approximately 14,000 active customers at the end of the first quarter of 2010! Among other news:

  • Qliktech hired 103 new employees in Q1 of 2011 and currently employed 883 people (a 43% increase year-over-year).
  • Qliktech signed a strategic alliance with Deloitte, starting with Netherlands and planning expansion of alliance to Deloitte worldwide.
  •  About 2 weeks ago Qliktech unveils one of the first HTML5-based full client application: Qlikview on iPad (free [user will need license to access Qlikview Server anyway] – and delivered it through the Safari mobile Web browser) – Qliktech claims that it is “every bit as rich as a native app.”

I guess most of DV Client applications should have HTML5 reincarnation soon… As a result of all these positive sound bites, Qliktech shares ended this week above $32, more than tripled in 9 months:


and I compared Qliktech’s relative growth in above Annotated Timeline chart with Microstrategy, TIBCO and Apple (yes, Qliktech is growing at least twice faster than … Apple). I cannot include Tableau in comparison, because Tableau Software is still … a private company.

Qliktech’s capitalization as of today, 4/30/11 is $2.5B, $1B more than Microstrategy and only twice less than TIBCO’s capitalization. I know at least 3 software vendors, who are focused only on BI and DV: Tableau (it is still a private company; BTW, Tableau 6.1 will be released soon) – growing faster (114% YoY- see it here) than anybody, Qliktech (share price has tripled in last 9 months) and Microstrategy (it’s share price almost doubled in last 9 months). I consider the dedication to DV and BI as very important for future success in DV market; for example TIBCO’s Spotfire is only one of 50+ TIBCO’s products… and it dangers the future of one of the most advanced and mature DV products – Spotfire (version 3.3 is coming soon) .

One of reasons for Qliktech growth is its 1000+ partners and extensive Partner Programs for OEM Partners, Solution Providers, Business Consultants and System Integrators. Those overdeveloped Partner Programs required mandatory commitments from Partners in terms of Revenue Targets, Membership Fees, Qlikview Certifications and Minimum number of Trained employees. Lately Qliktech unreasonably raised those requirements and it may backfire and slowdown Qliktech growth and will help competitors like Tableau (Tableau actually opposite to Qliktech: their partnership program is underdeveloped – in my opinion – and requires big improvements) and recently Microstrategy (which seems learning from own and competitors mistakes and catching up lately).

Update 3 months later:

in Q2 of 2011 Qliktech reached 21000 customers worldwide (it means almost 1 million licensed users), $74 Millions in revenue (45% over Q2 2o1o); 1000 full time employes (400+ more compare with Q2 2010), $2.4B Market Capitalization and guess what – $2.2  Million of lost!

Permalink: https://apandre.wordpress.com/2011/04/30/good-week-for-qliktech/

Microstrategy is a famous and BI-dedicated company, operating for 22+ years, recently released Visual Insight (as part of the release of Microstrategy 9.2 this week) and joint the DV race. A couple of years ago, I advised to some local company in terms of choosing Data Visualization Partner and final 3 choices were Qlikview, Spotfire and Microstrategy. Microstrategy was most competitive pricing-wise, but their Data Visualization functionality was not ready yet. They are ready now, see it here (from webcast this week):

Visual Insight as part of Microstrategy 9.2 targets so called “self-service BI”, and transition (they acknowledged that) from “old BI” (tabular reports: published static and OLAP reports) to “new BI” (Data Visualization and Dashboards), from Desktop to Mobile Clients (that is a forward looking statement for sure), from Physical to Cloud.

Microstrategy is claiming that Visual Insight allows to visualize Data in 30 minutes (that is good to know, but DV Leaders already have it for a while, welcome to the club!) compare with 30 days for the same process with “traditional BI”:

(I am saying this for 6 years now and on this blog since inception of it; does it mean that old BI is useless now and too pricey? Microstrategy presenters saying that answer is yes! and I want to thank Microstrategy for the validation of my 6-years old conclusion). For full set of Microstrategy 9.2 slides click here.

Microstrategy 9.2 has a full BI product portfolio, fast in-memory Data Engine, free mobile and tablet clients, has even Free Reporting Suite . Microstrategy (like Qliktech, Tableau and Visokio) is completely focused on Business Intelligence and Data Visualization functionality unlike its giant competitors like SAP, IBM, Oracle and Microsoft.

Update 9/27/11. MIcrostrategy released free Cloud Personal edition, based on Visual Insight, see it for yourself here:

Since many people will use Excel regardless of how good other BI and DV tools are, I am regularly comparing abilities of Excel to solve Data Visualization problems I discussed on this site. In most cases Excel 2003 is completely inappropriate and obsolete (especially visually), Excel 2007 is good only for limited DV tasks like Infographics, Data Slides, Data Presentations, Static Dashboards and Single-Chart Visualizations. Excel 2010 has some features relevant to Data Visualizations, including one of the best columnar in-memory databases (PowerPivot as free add-in), an ability to synchronize multiple Charts through slicers, a limited ability to drilldown data using slicers and even the support for both 64-bit and 32-bit. However, when comparing with Qlikview, Spotfire and Tableau the Excel 2010 feels like a stone-age tool or at least 2 generation behind as far as Data Visualization (and BI) is a concern…

That was my impression until I started to use the Excel Plugin, called Visubi (from company with the same name, see it here ). Suddenly my Excel 2003 and Excel 2007 (I keep them for historical purposes) started to be almost as capable as Excel 2010, because Visubi adding to all those versions of Excel a very capable columnar in-memory database, slicers and many features you cannot find in Excel 2010 and PowerPivot and in addition is greatly improving the functionality of Excel PivotTables and Tables! Vizubi enables me to read (in addition to usual data sources like ODBC, CSV, XLS, XLSX etc.) even my QVD files (Qlikview Data files)! Visubi, unlike PowerPivot, will create Time Dimension(s) the same way as SSAS does. All above means that users are not forced to migrate to Office 2010, but they will have many PowerPivot features with their old version of Excel. In addition Vizubi added to my Excel tables and Pivots uniques feature: I can easily switch back and forth between Table and PivotTable presentation of my data.

Most important Visubi’s feature is that all Vizubi’s tables and pivots are interactive and each piece of data is clickable and enables me to drill down/up/through my entire dataset:

It is basically equivalent or exceeded the drilldown ability of Qlikview, with one exception: Qlikview allows to do it through charts, but Vizubi does it through Tables and PivotTables. Visubi enables Excel user creates large databases with millions of rows (e.g. test database has 15 millions of rows) and enables ordinary users (non-developers) easily create Tables, Reports, Charts, Graphs and Dashboards with such database – all within familiar Excel environment using easy Drag-and-Drop UI:

Vizubi’s Database(s) enables users to share data over central datastore, while keeping Excel as a personal desktop DV (or BI) client. See Vizubi videos here and tutorials here.

Vizubi is a small (15 employees) profitable Italian company and it is a living prove that size does not matter – Vizubi did something extremely valuable and cool for Excel users that giant Microsoft failed to do for many years, even with PowerPivot. Prices for Vizubi is minimal considering the value it adds to Excel: between $99 and &279, depends on the version and the number of seats (discounts are available, see it here ).

Vizubi is not perfect (they just at version 1.21, less then one year old product), for example I wish they will support a graphical drilldown like Qlikview does (outlining rectangles right on Charts and then instant selection of appropriate subset of data ), a web client (like Spotfire) and web publishing for their functionality (even Excel 2010 supports Slicers on a web in Office Live environment), 64-bit Excel (32-bits is so 20th century), the ability to read and use SSAS and PowerPivot directly (like Tableau does), some scripting (Javascript or VBScript like Qlikview) and”formula”  language (like PowerPivot with DAX) etc.

I suggest to review these articles about Vizubi: in TDWI by Stephen Swoyer and relatively old article  from Marco Russo at SQLBlog .

Permalink: https://apandre.wordpress.com/2011/04/10/visubi/

Heritage Provider Network is offering a cool $3 millions in prize money for the development of an algorithm that can best predict how often people are likely to be sent to the hospital. Jonathan Gluck — senior executive at Heritage — said the goal of the competition is to create a model that can “identify people who can benefit from additional services,” such as nurse visits and preventive care. Such additional services could reduce health care spending and cut back on excessive hospitalizations, Gluck said.

The algorithm contest, the largest of its kind so far, is an attempt (also see Slate article here) to help find the best answers to complicated data-analysis questions. Previous known was the $1 million Netflix Inc. prize awarded in 2009 for a model to better predict what movies people would like. In 2009, a global team of seven members consisting of statisticians, machine-learning experts and computer engineers was awarded the $1 Million contest prize and Netflix replaced its legacy recommendation system with the team’s new algorithm (2nd Netflix’s competition was stopped by FTC and lawyers). I personally think that this time Data Visualization will be a large part of winning solution.

The competition — which will be run by Australian startup firm Kaggle — begins on April 4 and will be open for about two years. Contestants will have access to de-identified insurance claims data to help them develop a system for predicting the number of days an individual is likely to spend in a hospital in one year. Kaggle spent months streamlining claims data and removing potentially identifying information, such as names, addresses, treatment dates and diagnostic codes. Teams will have access to three years of non-identifiable healthcare data for thousands of patients.
The data will include outpatient visits, hospitalizations, medication claims and outpatient laboratory visits, including some test results. The data for each de-identified patient will be organized into two sections: “Historical Data” and “Admission Data.” Historical Data will represent three years of past claims data. This section of the dataset will be used to predict if that patient is going to be admitted during the Admission Data period. Admission Data represents previous claims data and will contain whether or not a hospital admission occurred for that patient; it will be a binary flag.

The training dataset includes several thousand anonymized patients and will be made available, securely and in full, to any registered team for the purpose of developing effective screening algorithms. The quiz/test dataset is a smaller set of anonymized patients. Teams will only receive the Historical Data section of these datasets and the two datasets will be mixed together so that teams will not be aware of which de-identified patients are in which set.

Teams will make predictions based on these data sets and submit their predictions to HPN through the official Heritage Health Prize web site. HPN will use the Quiz Dataset for the initial assessment of the Team’s algorithms. HPN will evaluate and report back scores to the teams through the prize website’s leader board.

Scores from the final Test Dataset will not be made available to teams until the accuracy thresholds are passed. The test dataset will be used in the final judging and results will be kept hidden. These scores are used to preserve the integrity of scoring and to help validate the predictive algorithms. You can find more about Online Testing and Judging here.

The American Hospital Association estimates that more than 71 million people are admitted to the hospital each year, and that $30 Billion is spent on unnecessary admissions.

Pagos released this week SpreadsheetWEB 3.2 (PSW for short) with new Data Visualization features (Pagos Data Visualizer or PDV for short). Among those features is an ability to drill-down any Visible Data through synchronized filters, which immediately made the SpreadsheetWEB a player in Data Visualization Market.

Tools like Tableau, Qlikview or Spotfire allow people to visualize data, but have very limited ability to collect and update data. PSW (Pagos SpreadsheetWEB), on other hand, since versions 1.X was able to convert any Excel Spreadsheet into Web Application and Web-based Data Collector, to save collected data into SQL Server (including latest SQL Server 2008 R2) Database, and to Report or Visualize the Data online through SaaS web-based spreadsheet, which looks and behaves as Excel Spreadsheet! SpreadsheetWEB has unique ability to collect data in a Batch Process and run large datasets against SpreadsheetWEB application. This video demonstrates data collection and data management and collaborations utilizing workflow capabilities and SpreadsheetWEB Control Panel interface. SpreadsheetWEB can use Web-Service as Data Source (like Excel does) and allows web-based spreadsheets to function as Web Service too:

One of the reasons why most people still use and like Excel as a BI tool is that they can use many of the built-in worksheet formulas to process data in real-time while filtering the dashboard. SpreadsheetWEB converts those formulas and can execute them on the server. Database-driven SpreadsheetWEB applications support most features in Excel, including worksheet formulas, 333+ Excel functions, formatting, 33+ types of Excel charts as well as Sparklines,

also see video here:

as well as pivot tables, validation, comments, filters and hyperlinks, while almost completely eliminating the need for application and database developers, as well as need for IT services. Basically if person knows Excel, than he knows how to use SpreadsheetWEB. SpreadsheetWEB (both 64-bit and 32-bit) has HTML Editor and Scripting Support (JavaScript), similar to what macros do for Excel (be aware that it is not port of VBA):

Among 3 DV Leaders only Tableau is able to read Microsoft SQL Server Analysis Services (SSAS) data sources, which is a must for long-term success in Visual Analytics market. SpreadhseetWEB has this functionality the same way as Excel does and therefore ahead of Qlikview and Spotfire in this extremely important department. Among other advanced Data Visualization Features SpreadsheetWEB supports Maps in Dashboards

and multi-page Dashboard reports. I like Version Control for applications and Server Monitoring features – they can be very attractive for enterprise users. SpreadsheetWEB does not require SharePoint Server to execute Excel workbooks on the server. Pagos developed proprietary spreadsheet technology to achieve that independence from SharePoint Server (I personally consider SharePoint as a Virus). This makes Pagos very attractive to cost conscious small to medium size organizations. Installing SpreadsheetWEB only requires Windows Server and Microsoft SQL Server. In addition, SpreadsheetWEB works with free SQL Express Edition, which is an additional savings for Customers with small datasets.

For advanced Data Visualization functionality, Pagos established the OEM partnership with TIBCO and integrates SpreadsheetWEB with TIBCO Spotfire Analytic Platform. For advanced SaaS features, including strictest security and hosting requirements and SAS70 Compliance, Pagos partners with Rackspace.

SpreadsheetWEB is one of the few players in the market that offer Software-as-a-Service (SaaS) licensing along with traditional server licensing. Pagos has very attractive SaaS fees and extremely competitive pricing for those who want to buy own SpreadsheetWEB server: $4900 per SpreadsheetWEB server for 50 named users and 25 web applications and dashboards; that price at least 10 times better than prices from Qlikview, Spotfire and Tableau. Pagos provides 44+ Video Tutorials, 53+ online Demos, free non-expiring trial and Wiki-based full Documentation for SpreadsheetWEB, so people can review, browse and evaluate SpreadsheetWEB way before they will buy it.

Pagos is in BI business since 2002, profitable and fully self-funded since inception, with hundreds of customers. Pagos has other advanced BI-related products, like SpreadsheetLIVE (it offers a fully featured spreadsheet application environment within a web browser) and Pagos Spreadsheet Component (allows software developers to create web and desktop applications that can read, execute, and create Excel spreadsheets without requring Microsoft Excel). If you will compare SpreadsheetWEB with Microsoft’s own attempt to webify Excel and Microsoft’s own Long List of Unsupported Excel features, you can easily appreciate the significance of what Pagos achieved!

Permalink: https://apandre.wordpress.com/2011/03/13/spreadsheetweb/

I never saw before when one man moved from one company to another, then 46+ people will almost immediately comment on it. But this is what happened during last few days, when Donald Farmer, the Principal Program Manager for Microsoft BI Platform for 10 years, left Microsoft for Qliktech. Less than one year ago, Donald compared Qlikview and PowerPivot and while he was respectful to Qlikview, his comparison favored PowerPivot and Microsoft BI stack. I can think/guess about multiple reasons why (and I quote him: “I look forward to telling you more about this role and what promises to be a thrilling new direction for me with the most exciting company I have seen in years”) he did it, for example:

  • Microsoft does not have a DV Product (and one can guess that Donald wants to be the “face” of the product),
  • Qliktech had a successful IPO and secondary offering (money talks, especially when 700-strong company has $2B market capitalization and growing),
  • lack of confidence in Microsoft BI Vision (one can guess that Donald has a different “vision”),
  • SharePoint is a virus (SharePoint created a billion dollar industry, which one can consider wasted),
  • Qlikview making a DV Developer much more productive (a cool 30 to 50 times more productive) than Microsoft’s toolset (Microsoft even did not migrate the BIDS 2008 to Visual Studio 2010!),
  • and many others (Donald said that for him it is mostly user empowerment and user inspiration by Qlikview – sounds like he was underinspired with Microsoft BI stack so is it just a move from Microsoft rather then move  to Qliktech? – I guess I need a better explanation),

but Donald did explain it in his next blog post: “QlikView stands out for me, because it not only enables and empowers users; QlikView users are also inspired. This is, in a way, beyond our control. BI vendors and analysts cannot prescribe inspiration“. I have to be honest – and I repeat it again – I wish a better explanation… For  example, one my friend made a “ridiculous guess” that Microsoft sent Donald inside Qliktech to figure out if it does make sense to buy Qliktech and when (I think it is too late for that, but at least it is an interesting thought: good/evil  buyer/VC/investor will do a “due diligence” first, preferably internal and “technical due diligence” too) to buy it and who should stay and who should go.

I actually know other people recently moved to Qliktech (e.g. from Spotfire), but I have a question for Donald about his new title: “QlikView Product Advocate”. According to http://dictionary.reference.com/ the Advocate is a person who defends, supports and promotes a cause. I will argue that Qlikview does not need any of that (no need to defend it for sure, Qlikview has plenty of Supporters and Promoters); instead Qlikview needs a strong strategist and visionary

(and Donald is the best at it) who can lead and convince Qliktech to add new functionality in order to stay ahead of competition with at least Tableau, Spotfire and Microsoft included. One of many examples will be an ability to read … Microsoft’s SSAS multidimensional cubes, like Tableau 6.0 and Omniscope 2.6 have now.

Almost unrelated – I updated this page:  https://apandre.wordpress.com/market/competitors/qliktech/

Permalink: https://apandre.wordpress.com/2011/01/09/farmer_goes_2_qlikview/

Microsoft reused its patented VertiPaq column-oriented DB technology in upcoming SQL Server 11.0 release by introducing columnstore indexes, where each columns stored in separate set of disk pages. Below is a “compressed” extraction from Microsoft publication and I think it is very relevant to the future of Data Visualization techologies. Traditionally RDBMS uses “row store” where

heap or a B-tree contains multiple rows per page. The columns are stored in different groups of pages in the columnstore index. Benefits of this are:

  • only the columns needed to solve a query are fetched from disk (this is often fewer than 15% of the columns in a typical fact table),
  • it’s easier to compress the data due to the redundancy of data within a column, and
  • buffer hit rates are improved because data is highly compressed, and frequently accessed parts of commonly used columns remain in memory, while infrequently used parts are paged out.

“The columnstore index in SQL Server employs Microsoft’s patented Vertipaq™ technology, which it shares with SQL Server Analysis Services and PowerPivot. SQL Server columnstore indexes don’t have to fit in main memory, but they can effectively use as much memory as is available on the server. Portions of columns are moved in and out of memory on demand.” SQL Server is the first major database product to support a pure Columnstore index. Columnstore recommended for fact tables in DW in datawarehouse, for large dimensions (say with more than 10 millions of records) and any large tables designated to be used as read-only.

“In memory-constrained environments when the columnstore working set fits in RAM but the row store working set doesn’t fit, it is easy to demonstrate thousand-fold speedups. When both the column store7and the row store fit in RAM, the differences are smaller but are usually in the 6X to 100X range for star join queries with grouping and aggregation.” Your results will of course depend on your data, workload, and hardware. Columnstore index query processing is most heavily optimized for star join queries. OLTP-style queries, including point lookups, and fetches of every column of a wide row, will usually not perform as well with a columnstore index as with a B-tree index.

Columnstore compressed data with a factor of 4 to a factor of 15 compression with different fact tables. The columnstore index is a secondary index; the row store is still present, though during query processing it is often not need, and ends up being paged out. A clustered columnstore index, which will be the master copy of the data, is planned for the future. This will give significant space savings.

Tables with columnstore indexes can’t be updated directly using INSERT, UPDATE, DELETE, and MERGE statements, or bulk load operations. To move data into a columnstore table you can switch in a partition, or disable the columnstore index, update the table, and rebuild the index. Columnstore indexes on partitioned tables must be partition-aligned. Most data warehouse customers have a daily, weekly or monthly load cycle, and treat the data warehouse as read-only during the day, so they’ll almost certainly be able to use columnstore indexes.You can also create a view that uses UNION ALL to combine a table with a column store index and an updatable table without a columnstore index into one logical table. This view can then be referenced by queries. This allows dynamic insertion of new data into a single logical fact table while still retaining much of the performance benefit of columnstore capability.

Most important for DV systems is this statement: “Users who were using OLAP systems only to get fast query performance, but who prefer to use the T-SQL language to write queries, may find they can have one less moving part in their environment, reducing cost and complexity. Users who like the sophisticated reporting tools, dimensional modeling capability, forecasting facilities, and decision-support specific query languages that OLAP tools offer can continue to benefit from them. Moreover, they may now be able to use ROLAP against a columnstore-indexed SQL Server data warehouse, and meet or exceed the performance they were used to in the past with OLAP, but save time by eliminating the cube building process“. This sounds like Microsoft finally figured out of how to compete with Qlikview (technology-wise only, because Microsoft still does not have – may be intentionally(?) – DV product).

Permalink: https://apandre.wordpress.com/2010/12/03/columnstore-index/

SAP released HANA today which does in-memory computing with in-memory database. Sample appliance with 10 blades with 32 cores (using XEON 7500) each; sample (another buzzword: “data source agnostic”) appliance costs approximately half-million of dollars. SAP claimed that”Very complex reports and queries against 500 billion point-of-sale records were run in less than one minute” using parallel processing. SAP HANA “scales linearly” with performance proportional to hardware improvements that enable complex real-time analytics.

Pricing will likely be value based and that it is looking for an all-in figure of around $10 million per deal. Each deal will be evaluated based upon requirements and during the call, the company confirmed that each engagement will be unique (so SAP is hoping for 40-60 deals in pipeline).

I think with such pricing and data size the HANA appliance (as well as other pricey data appliances) can be useful mostly in 2 scenarios:

  • when it integrates with mathematical models to enable users to discover patterns, clusters, trends, outliers and hidden dependencies and
  • when those mountains of data can be visualized, interactively explored and searched, drilled-down and pivot…

8/8/11 Update: The 400 million-euro ($571 million) pipeline for Hana, which was officially released in June, is the biggest in the history of Walldorf, Germany-based SAP, the largest maker of business-management software. It’s growing by 10 million euros a week, co-Chief Executive Officer Bill McDermott said last month. BASF, the world’s largest chemical company, has been able to analyze commodity sales 120 times faster with Hana, it said last month. Russian oil producer OAO Surgutneftegas, which has been using Hana in test programs since February, said the analysis of raw data directly from the operational system made additional data warehouse obsolete.

Permalink: https://apandre.wordpress.com/2010/12/01/sap-hana/

Microsoft used to be a greatest marketing machine in software industry. But after loosing search business to Google and smartphone business to Apple and Google they lost their winning skills. It is clear now that this is also true in so called BI Market (Business Intelligence is just a marketing term).  Microsoft bought ProClarity and it disappeared, they released PerformancePoint Server and it is disappearing too. They have (or had?) the best BI Stack (SQL Server 2008 R2 and its Analysis Services, Business Intelligence Development Studio 2008 (BIDS), Excel 2010, PowerPivot etc.) and they failed to release any BI or Data Visualization Product, despite having all technological pieces and components. Microsoft even released Visual Studio 2010 without any support for BIDS and recently they talked about their Roadmap for BI and again – they delayed the mentioning of BIDS 2010 and they declared NO plans for BI or DV products! Instead they are talking about “new ad hoc reporting and data visualization experience codenamed “Project Crescent””!

And than they have a BISM model as a part of Roadmap: “A new Business Intelligence Semantic Model (BISM) in Analysis Services that will power Crescent as well as other Microsoft BI front end experiences such as Excel, Reporting Services and SharePoint Insights”.

Experience and Model instead of Product? What Microsoft did with PowerPivot is clear: they gave some users the reason to upgrade to Office 2010, and as a result, Microsoft preserved and protected (for another 2 years?) their lucrative Office business but diminished their chances to get a significant pie of $11B (and  growing 10% per year) BI Market. new BISM (Business Intelligence Semantic Model) is a clear sign of losing technological edge:

image

I have to quote (because they finally admitted that BIDS will be  replaced by BISM – when “Project Juneau” will be available): “The BI Semantic Model can be authored by BI professionals in the Visual Studio 2010 environment using a new project type that will be available as part of “Project Juneau”. Juneau is an integrated development environment for all of SQL Server and subsumes the Business Intelligence Development Studio (BIDS). When a business user creates a PowerPivot application, the model that is embedded inside the workbook is also a BI Semantic Model. When the workbook is published to SharePoint, the model is hosted inside an SSAS server and served up to other applications and services such as Excel Services, Reporting Services, etc. Since it is the same BI Semantic Model that is powering PowerPivot for Excel, PowerPivot for SharePoint and Analysis Services, it enables seamless transition of BI applications from Personal BI to Team BI to Organizational (or Professional) BI.

Funniest part of this quote above that Microsoft is honestly believe that SharePoint is not a Virus but a viable Product and it will escape the fate of its “step-brother” – PerfromancePoint Server. Sweet dreams! It is clear that Microsoft failed to understand that Data Visualization is the future of BI market and they keep recycling for themselves the obvious lie “Analysis Services is the industry leading BI platform in this space today“! Indirectly they acknowledged it in a very next statement : “With the introduction of the BI Semantic Model, there are two flavors of Analysis Services – one that runs the UDM (OLAP) model and one that runs the BISM model”. Hello?

Why we need 2 BI Models instead of 1 BI product? BIDS 2008 itself is already buggy and much less productive development environment than Qlikview, Spotfire and Tableau, but now Microsoft wants us to be confused with 2 co-existing approaches: OLAP and BISM? And now get this: “you should expect to see more investment put into the BISM and less in the UDM(OLAP)”!

Dirty Harry will say in such situation: “Go ahead, make my day!” And I guess that Microsoft  does not care that Apple’s  Market CAP is larger than Microsoft now.

Afterthought (looking at this from 2011 point of view): I am thinking now that I know why Donald Farmer left Microsoft 2 months after BISM announcement above.

p010: http://wp.me/pCJUg-7r

DV (Data Visualization) makes more sense when you trying to Visualize huge datasets, which indirectly implies the eventual need for DW (Data Warehouses) and DW appliances (DWA). Among pioneers for DWA we can name the Teradata . This was not a very hot area until 7/6/10, when EMC bought Greenplum with its own MPP architecture. On 9/20/10 IBM bought Netezza for $1.7B and DWA market became officially hot in anticipation of need of DV and BI users for a lot of DWA for their “big data”. Teradata claimed 2 years ago that Netezza are far behind performance-wise, but apparently IBM disagrees or does not care… Please note that Netezza,  before it was bought, pro-actively partnered with DV vendors, using them as a way to expand their market share and this points us to the future.

With “big data” buzz everywhere, I suspect a large wave of partnerships between DWA (EMC DCA (Data Computing Appliance), IBM, Teradata, Microsoft / DATAlegro, Oracle / Exadata, SAP ( HANA + Sybase IQ ) vendors, as well as vendors of virtual DWAs)  and DV vendors is coming in 2011. Data Visualization making DWA much more attractive for end users with huge datasets! Microsoft’s PDW was released on 11/9/10 and SAP HANA will be released in November 2010 too

p008: http://wp.me/sCJUg-dwa

BI and DV vendors do not want me to relax and keep releasing new stuff too often. I feel guilty now and I will (3+ months after it was released) comment on Spotfire 3.2 release soon. But today I have to comment on Cognos 10 Release (which will be available Oct. 30; everybody now does pre-announcement: 2 weeks ago Qlikview 10, yesterday BO4, today Cognos 10). I quote: “IBM acquired Cognos in early 2008 during a five year buying spree that saw it swallow over 24 analytics companies in five years for a total bill of US$14 billion”. Rob Ashe, general manager for BI at IBM, said: ““Analytics is a key part of our 2015 roadmap. Last year, analytics contributed $9 billion to our revenues, and we expect to see that grow to $16 billion in 2015.”

The Cognos 10 embeds SSPS and Lotus Connections, supports SaaS, active/interactive reports via email (no need to install anything), mobile devices such as iPhones, iPads and BlackBerrys (as well as Symbian phones, and Windows Mobile devices), real-time updates, has “modern” Web 2.0 user interface. Cognos TM1 (from Applix) is a multidimensional, 64-bit, in-memory OLAP engine which provides fast performance for analyzing complex and sophisticated models, large data sets and even streamed data.

Personally I think Cognos 10 compares favorably against BO4, SAS 9.2, OBIEE 11g , but all 4 have at least 2 common problems: they are all engaged too much with Java and they are far (of Qlikview, Spotfire, Omniscope, Tableau etc.) behind in Data Visualization

n006: http://wp.me/pCJUg-4Z

Business Objects 4.0 will be available this  (2010) year” – SAP teases own customers at ASUG. It became a habit for SAP – to say something about a product they did not release yet. For example they did pre-announcement of HANA (in-memory analytics appliance) in May 2010, see http://www.infoworld.com/d/applications/sap-build-new-in-memory-database-appliances-392 and now they are saying that HANA will be released in November 2010: http://www.infoworld.com/d/applications/saps-in-memory-analytics-boxes-set-november-release-117 . It is very funny to see how 3 (SAP, IBM, Oracle) or 4 (if  you include the mindshare leader SAS) BI behemoths trying to compete (using money instead of creativity) with DV leaders like Qlikview and Spotfire who has in-memory columnar DB for years. E.g. IBM recently bought Netezza, SSPS and Applix and trying to marry Applix with Cognos. Or Oracle (after buying Sun) releasing Exadata and Exalogic to compete with… IBM’s Netezza and SAP’s HANA. SAP actually owns now (after they recently bought Sybase) the best collection of BI and DV-related technologies, like best columnar DB Sybase IQ (ok, Vertica too, but Qlikview, PowerPivot and Spotfire have it in-memory).

Back to BO4: it will be 64-bit only, Desktop Intelligence will not be included in this release, BO4 will be more dependent on Java (SAP, IBM, Oracle and SAS –  all 4 making a strategic mistake by integrating their product with dying Java), BO4 will have “data federation”, BO4 will be integrated with SAP Portfolio (e.g. NetWeaver), Bo4 has now multi-dimensional analytical ability,
SAP Explorer allows in-memory Analytics etc. It took SAP 4+ months from pre-anouncement to release of BO4 – I guess they learn from
Microsoft (I am not sure how it helps).

Update as of 7/27/11: BI 4.0 still not released yet and SAP is planning to release it now in August 2011, basically 10 months later then it was pre-anounced! Among other updates: on 7/25/11 SAP released interesting video with Demo:

Update as of 8/31/11: It took SAP 11 months from pre-announcement of BO4 to officially release it, see http://blogs.sap.com/analytics/2011/08/31/update-on-sap-businessobjects-bi-4-0-general-availability/   SAP said today: “Based on efforts over the last several weeks, BI 4.0 is targeted to become generally available starting September 16, 2011.” Also “For customers and partners currently using BI 4.0, new eLearning tutorials are now available on the SAP Community Network. Check out the latest tutorials and take advantage of the new capabilities BI 4.0 has to offer.” It is very funny and very sad RELEASE process.

Enterprise Deployment of SAP BO may look like this:


n005: http://wp.me/pCJUg-4o