July 18, 2014
While on Cape Cod this summer and when away from beach, I enjoyed some work-unrelated fun with Tableau. My beach reading included this article: http://www.theinformationlab.co.uk/2014/03/27/radar-charts-tableau-part-3/ by Andrew Ball and I decided to create my own Radar. When I show it to coworkers later, they suggested to me to publish it (at least the fun with Polygons, Path and Radars) on my blog. I may reuse this Radar chart for our internal Web Analytics.
Natural Order of Points and Segments in Line.
Many visualization tools will draw the line chart, its datapoints and connecting line segments between datapoints in natural progressing order – repainting them from left to right (horizontal ordering by Axis X) or from bottom to upside
(vertical ordering by Axis Y) or vice versa.
Path as the method to break the Natural Order.
Some demanding visualizations and users wish to break the natural repainting and drawing order and Tableau allows to do that by using the Path as the method to order the datapoints and line segments in Lines and Polygons. A Collection of increasing Ordering Numbers (Pathpoints) for each Datapoint in Line defined a Path for drawing and connecting datapoints and segments of that Line (or Polygon). Each Pathpoint can be predefined or calculated, depends on mplementation and business logic.
Changing the Natural Order can create “artificial” and unusual situations, when two or more datapoints occupying the same pixels on drawing surface but have very different Pathpoints (example can be a Polygon, when Line ends in the same point it starts) or when two or more Line Segments intersecting in the same Pixel on screen (example can be the Center of the letter X ).
Radar Chart has 2 parts: Radar Grid (background) and Radar Polygons (showing repetitive Data Patterns, if linear timeline can be collapsed into circular “timeline”). Radar Grid has Radials (with common Center) and Concentric Rings.
Polygons optionally can be filled with (transparent) color. For future Discussion let’s use the RMax as the maximal possible distance between the Center of Radar Grid (in case of Radar Grid) or the Center of Radar Polygon (in case of Radar
Polygon) and the most remote Datapoint shown in Radar Grid or Polygon respectively. We will use the “normalized” statistics of Visits to typical Website to visualize the hourly and daily (by day of the week) patterns of Web Visitations. By
normalization we mean the removal of insignificant deviations from “normal” hourly and daily amounts of Web Visits. For complete obfuscation we will assume for Demo purposes that RMax = 144.
Radar Radial Grid.
Radial Grid contains a few Radiuses (equidistant from each other) and we will draw each Radius as 3-point line where Starting and Ending points of each line are identical to each other and collocated with the Center of Radar. For Demo Web Visitation Radar we will use Radial Grid with 8 Radiuses, corresponding to the following hours of the complete 24-hours day: 0, 3, 6, 9, 12, 15, 18, 21:
For example see the Radius, corresponding to HOUR = 3 (below in light brown, other Radiuses greyed out on that image):
And for that Radius we are using (redundantly) the following 3 datapoints:
Concentric Rings for Radar Grid.
For Demo Radar we will use 4 Concentric Rings, corresponding to 25%, 50%, 75% and 100% levels of maximum visitation per hour:
Each ring is a line with 25 datapoints, where Starting and Ending Points collocated/equal. For example, dataset for external Ring (red line above) looks like this:
When Radials and Concentric Rings collocated and overlaid they represent the Radar Grid, ready to be a background for Radar Chart:
For Demo purposes we use only 2 Polygons – one (largest) representing average Hourly Visits during Weekday and 2nd Polygon representing average Hourly Visits during Weekend day. For Websites which I observed the minimum number of visits happened around 1 AM, so you will see both Polygons are slightly rotated clockwise and slightly shifted up from the Center of Radar Grid to reflect the fact that the minimum number of visitors (even around 1 AM) is slightly more then 0. Each Radar Polygon (in our Demo) has 25 Data Points with Starting and Ending Points collocated at 1AM. Here is a Weekday Polygon, overlaid with Radar Grid:
Here are the data for Weekday Polygon:
Here is a Polygon for Weekend day, overlaid with Radar Grid:
When Radar Grid and Radar Polygons overlaid (Polygons transparent but on top of
Grid) we will get the Radar Chart. Please note that Centers of Radar Grid and Radar
Polygons can have different locations:
I published Tableau workbook with this Demo Radar Chart and Radar Data here:
January 19, 2014
Leave a Comment
This is a repost from Data Visualization Consulting Page.
Visitors of this blog generated a lot of requests for my Data Visualization “Advice” (small projects for a few hours or days, no NDA [Non-Disclosure Agreement] involved), for Data Visualization Consulting projects (a few weeks or months; I tend to avoid the NDAs as they can interfere with my blogging activities) and even for Full-time work (for example my latest full-time job I got because my employer often visited and read my blog; NDA needed).
Additionally, sometimes I am doing free-of-charge work, if involved projects are short, extremely interesting for me and beneficial for my Data Visualization Blog, like this project:
Obviously all these projects can be done only when I have spare time either from full-time work and/or other projects, duties and activities.
I also cannot relocate or travel, so I can do it mostly from my home office – telecommuting (RDP, Skype, phone, WebEx, GoToMeeting etc.) or if client is local to Massachusetts, then sometime I can visit Client’s site, see below the Map of my Local “Service Area” – part of Middlesex County between Routes 495, 3 and 20 – where I can commute to Client’s Location (please click on map below to enlarge the image) :
If I do have time for short-term advisory projects (from 2 hours to 2 weeks), clients usually pay by the highest rate, similar to what Qliktech, Spotfire, Tableau or IBM charging for their Consulting Services (I consider my consulting as better service than theirs…). If you will go to this thread on Tableau Community:
http://community.tableausoftware.com/thread/127338 then you will find these Indicative Rates for Consulting Tableau Work (Qlikview and Spotfire Rates are very similar):
Low $125, Max $300, Average around $175 per hour.
Here are the most popular requests for my Advisory work:
- Visual Design and Architectural Advice for Monitoring or Operational Dashboard(s);
- Review of Data Visualization Work done by my Clients;
- Prototyping of Data Visualizations (most requested by my visitors);
- My opinion on Strengths and Weaknesses of Data Visualization Vendor/Product, requested by trader, portfolio or hedge fund manager(s)
- Advice about what Hardware to buy (say to get the most from Tableau License client has);
- Advice what Charts and Filters to use for given Dataset and Business Logic;
- Technical Due Diligence on Data Visualization Startup for Venture Capitalists investing into that Start-up.
For mid-size projects (from 2 weeks to 6 months) clients getting a “Progressive” discount – the longer the project then the larger the discount. Here are the most popular requests for my Consulting Data Visualization Work:
- Comparing Data Visualization Product vs. Other Visualization Product for specific Client’s needs and projects;
- Comparing Clients’s Visualization Product vs. Competitor(s) Visualization Product (most requested);
- Benchmarking one or more Visualization Product(s) vs. specific data and application logic.
- Managed Clients migration of their Reporting and Analytical IT Infrastructure from obsolete BI Platforms like Business Objects, Cognos and Microstrategy to modern Data Visualization Environments like Tableau, Qlikview and Spotfire.
Full-time work (1 year or more engagements) is not exactly a Consulting but Full-time job when clients asking me to join their company. These jobs are similar to what I had in the past: Director of Visual Analytics, Data Visualization Director, VP of Data Visualization, Principal Data Visualization Consultant, Tableau Architect etc. Here are samples of full-time projects:
- Created, Maintained and Managed the Data Visualization Consulting Practices for my company/employer;
- Led the growth of Data Visualization Community (the latest example – 4000 strong Tableau Community) with own Blog, Portal and User Group behind the corporate firewall, created Dozens of near-real-time Monitoring Dashboards for Analytical and Data Visualization Communities;
- Designed and Implemented myself hundreds of Practical Data Visualizations and Visual Reports, which led to discovery of trends, outliers, clusters and other Data Patterns, Insights and Actions;
- Created hundreds of Demos, Prototypes and Presentations for Business Users;
- Designed Data Visualization Architecture and Best Practices for Dozen of Analytical Projects;
- Significantly improved the Mindshare and increased the Web Traffic to website of my company, Created and Maintained the Data Visualization blog for it.
You can find more observations about relationship between Full-Time salary and Hourly Rate for consulting in my previous post (from 6 months ago) here: https://apandre.wordpress.com/2013/07/11/contractors-rate/
January 5, 2014
Data Visualization readings – last 4 months of 2013.
(time to read is shrinking…)
0. The Once and Future Prototyping Tool of Choice
1. Block by Block, Brooklyn’s Past and Present
2. Data Visualization and the Blind
3. WHY ABRAHAM LINCOLN LOVED INFOGRAPHICS
4. Old Charts
5. Back To Basics
6. In-Memory Data Grid Key to TIBCO’s Strategy
7. Submarine Cable Map
8. Interview with Nate Silver:
9. Qlikview.Next will be available in 2014
10. Importance of color?
11. Qlikview.Next has a gift for Tableau and Datawatch
12. (October 2013) Tableau posts 90% revenue gain and tops 1,000 staffers, files for $450 million secondary offering
13. The Science Of A Great Subway Map
14. SEO Data Visualization with Tableau
15. John Tukey “Badmandments”
- 91. NEVER plan any analysis before seeing the DATA.
- 92. DON’T consult with a statistician until after collecting your data.
- 94. LARGE enough samples always tell the truth
16. Thinking about proper uses of data visualization.
17. Big BI is Stuck: Illustrated by SAP BusinessObjects Explorer
18. IBM (trying to catch up?) bets on big data visualization
19. Site features draft designs and full views of the Treemap Art project (By Ben Shneiderman)
20. A Guide to the Quality of Different Visualization Venues
21. Short History of (Nothing) Data Science
22. Storytelling: Hans Rosling at Global Health – beyond 2015
23. DataWatch Quarterly Review: Rapid Growth Finally Materializing
24. QlikView Extension – D3 Animated Scatter Chart
25. SlopeGraph for QlikView (D3SlopeGraph QlikView Extension)
26. Recipe for a Pareto Analysis
27. Color has meaning
28. TIBCO’s Return To License Growth Frustratingly Inconsistent
29. Automated Semantics and BI
30. What is wrong with definition of Data Science?
31. Scientific data became so complex, we have to Invent new Math to deal with it
December 24, 2013
My Best Wishes for 2014 to all visitors of this Blog!
2013 was very successful year for Data Visualization (DV) community, Data Visualization vendors and for this Data Visualization Blog (number of visitors per grew from average 16000 to 25000+ per month).
From certain point of view 2013 was the year of Tableau – it went public, Tableau has now the largest Market Capitalization among DV Vendors (more than $4B as of Today) and its strategy (Data to the People!) became the most popular among DV users and it had (again) largest YoY revenue growth (almost 75% !) among DV Vendors. Tableau already employed more than 1100 people and still has 169+ job openings as of today. I wish Tableau to stay the Leader of our community and to keep their YoY above 50% – this will not be easy.
Qliktech is the largest DV Vendor and it will exceed in 2014 the half-billion dollars benchmark in revenue (probably closer to $600M by end of 2014) and will employ almost 2000 employees. Qlikview is one of the best DV product on market. I wish in 2014 Qlikview will create Cloud Services, similar to Tableau Online and Tableau Public and I wish Qlikview.Next will keep Qlikview Desktop Professional (in addition to HTML5 client).
I wish TIBCO will stop trying to improve BI or make it better – you cannot reanimate a dead horse; instead I wish Spotfire will embrace the approach “Data to the People” and act accordingly. For Spotfire my biggest wish is that TIBCO will spin it off the same way EMC did with VMWare. And yes, I wish Spofire Cloud Personal will be free and enabled to read at least local flat files and local DBs like Access.
2014 (or may be 2015?) can witness new, 4th DV player coming to competition: Datawatch bought recently Panopticon and if it will complete integration of all products correctly and add features which other DV vendors above already have (like Cloud Services), it can be very competitive player. I wish them luck!
Microsoft released in 2013 a lot of advanced and useful DV-related functionality and I wish (I recycling this wish for many years now) that Microsoft finally will package the most its Data Visualization Functionality in one DV product and add it to Office 20XX (like they did with Visio) and Office 365 instead of bunch of plug-ins to Excel and SharePoint.
It is a mystery for me why Panorama, Visokio and Advizor Solutions still relatively small players, despite all 3 of them having an excellent DV features and products. Based on 2013 IPO experience with Tableau may be the best way for them to go public and get new blood? I wish to them to learn from Tableau and Qlikview success and try this path in 2014-15…
For Microstrategy my wish is very simple – they are only traditional BI player who realised that BI is dead and they started in 2013 (actually before then 2013) a transition into DV market and I wish them all success they can handle!
I also think that a few thousands of Tableau, Qlikview and Spotfire customers (say 5% of customer base) will need (in 2014 and beyond) more deep Analytics and they will try to complement their Data Visualizations with Advanced Visualization technologies they can get from vendors like http://www.avs.com/
My best wishes to everyone! Happy New Year!
August 28, 2013
With releases of Spotfire Silver (soon to to be a Spotfire Cloud), Tableau Online and attempts of a few Qlikview Partners (but not Qliktech itself yet) to the Cloud and providing their Data Visualization Platforms and Software as a Service, the Attributes, Parameters and Concerns of such VaaS or DVaaS ( Visualization as a Service) are important to understand. Below is attempt to review those “Cloud” details at least on a high level (with natural limitation of space and time applied to review).
But before that let’s underscore that Clouds are not in the skies but rather in huge weird buildings with special Physical and Infrastructure security likes this Data Center in Georgia:
You can see some real old fashion clouds above the building but they are not what we are talking about. Inside Data Center you can see a lot of Racks, each with 20+ servers which are, together with all secure network and application infrastructure contain these modern “Clouds”:
Attributes and Parameters of mature SaaS (and VaaS as well) include:
- Multitenant and Scalable Architecture (this topic is too big and needs own blogpost or article). You can review Tableau’s whitepaper about Tableau Server scalability here: http://www.tableausoftware.com/learn/whitepapers/tableau-server-scalability-explained
- SLA – service level agreement with up-time, performance, security-related and disaster recovery metrics and certifications like SSAE16.
- UI and Management tools for User Privileges, Credentials and Policies.
- System-wide Security: SLA-enforced and monitored Physical, Network, Application, OS and Data Security.
- Protection or/and Encryption of all or at least sensitive (like SSN) fields/columns.
- Application Performance: Transaction processing speed, Network Latency, Transaction Volume, Webpage delivery times, Query response times
- 24/7 high availability: Facilities with reliable and backup power and cooling, Certified Network Infrastructure, N+1 Redundancy, 99.9% (or 99.99% or whatever your SLA with clients promised) up-time
- Detailed historical availability, performance and planned maintenance data with Monitoring and Operational Dashboards, Alerts and Root Cause Analysis
Disaster recovery plan with multiple backup copies of customers’ data in near real time at the disk level, a
multilevel backup strategy that includes disk-to-disk-to-tape data backup where tape backups serve as a secondary level of backup, not as their primary disaster recovery data source.
- Fail-over that cascades from server to server and from data center to data center in the event of a regional disaster, such as a hurricane or flood.
While Security, Privacy, Latency and Hidden Cost usually are biggest concerns when considering SaaS/VaaS, other Cloud Concerns surveyed and visualized below. Recent survey and diagram are published by Charlie Burns this month:
As you see above, Rack in Data Center can contain multiple Servers and other devices (like Routers and Switches, often redundant (at least 2 or sometimes N+1). Recently I designed the Hosting Data VaaS Center for Data Visualization and Business Intelligence Cloud Services and here are simplified version of it just for one Rack as a Sample.
You can see redundant network, redundant Firewalls, redundant Switches for DMZ (so called “Demilitarized Zone” where users from outside of firewall can access servers like WEB or FTP), redundant main Switches and Redundant Load Balancers, Redundant Tableau Servers, Redundant Teradata Servers, Redundant Hadoop Servers, Redundant NAS servers etc. (not all devices shown on Diagram of this Rack):
July 11, 2013
I got many questions from Data Visualization Blog’s visitors about differences between compensation for full-time employees and contractors. It turned out that many visitors are actually contractors, hired because of their Tableau or Qlikview or Spotfire skills and also some visitors consider a possibility to convert to consulting or vice versa: from consulting to FullTimers. I am not expert in all these compensation and especially benefits-related questions, but I promised myself that my blog will be driven by vistors’s requests, so I google a little about Contractor vs. Full-Time worker compensation and below is brief description of what I got:
Federal Insurance Contribution Act mandates Payroll Tax splitted between employer (6.2% Social Security with max $$7049.40 and 1.45% Medicare on all income) and employee, with total (2013) as 15.3% of gross compensation.
In addition you have to take in account employer’s contribution (for family it is about $1000/per month) to medical benefits of employee, Unemployment Taxes, employer’s contribution to 401(k), STD and LTD (short and long term disability insurances), pension plans etc.
I also added into my estimate of contractor rate the “protection” for at least 1 month GAP between contracts and 1 month of salary as bonus for full-time employees.
Basically the result of my minimal estimate as following: you need to get as a contractor the rate at least 50% more than base hourly rate of the full-time employee. This base hourly rate of full-time employee I calculate as employee’s base salary divided on 1872 hours: 1872 = (52 weeks*40 hours – 3 weeks of vacation – 5 sick days – 6 holidays) = 2080 hours – 208 hours (Minimum for a reasonable PTO, Personal Time Off) = 1872 working hours per year.
I did not get into account any variations related to the usage of W2 or 1099 forms or Corp-To-Corp arrangements and many other fine details (like relocation requirements and overhead associated with involvement of middlemen like headhunters and recruiters) and differences between compensation of full-time employee and consultant working on contract – this is just a my rough estimate – please consult with experts and do not ask me any questions related to MY estimate, which is this:
- Contractor Rate should be 150% of the base rate of a FullTimer
In general, using Contractors (especially for business analytics) instead of Full-timers is basically the same mistake as outsourcing and off-shoring: companies doing that do not understand that their main assets are full-time people. Contractors are usually not engaged and they are not in business to preserve intellectual property of company.
For reference see Results of Dr. Dobbs 2013 Salary Survey for Software Developers which are very comparable with salary of Qlikview, Tableau and Spotfire developers and consultants (only in my experience salary of Data Visualization Consultants are 10-15% higher then salaries of software developers):
This means that for 2013 the Average rate for Qlikview, Tableau and Spotfire developers and consultants should be around 160% of the base rate of a average FullTimer, which ESTIMATES to Effective Equivalent Pay to Contractor for 1872 hours per Year as $155,200 and this is only for average consultant... If you take less then somebody tricked you, but if you read above you already know that.