My little “Big Data” story

I recently attended an internal conference called CA Exchange in Orlando, Florida.  It’s for CA Technologies employees in engineering and product management to gather, meld minds, network and make our solutions better.  It was a great event for us “CAer’s” to share some pretty innovative tools and solutions.  On a whim, I attended a few of the “Big Data” sessions at this conference to get a better understanding of what our teams internally are working towards to help us make sense of our customer data.

It just so happens that I had a little Big Data problem/story to share myself.  One where we never knew where the information could take us until we began to look at it on different levels.

One of the products I manage is called CA AppLogic.  It’s a turn-key cloud platform enabling a true software defined datacenter.  CA AppLogic is used by many Managed Service Providers (MSP’s) who need to build and deliver cloud services for their customers quickly, efficiently and reliably.  Now, I won’t get into the detail on everything the technology can do, but the point I’m getting to is this:  AppLogic runs applications in the cloud but it’s an on premise solution (AppLogic must run on bare metal).  So what kind of data can this cloud platform gather?  About 200 data points and growing.  What makes the data both valuable and a problem is it’s frequency and depth:

AppLogic gathers data from each of it’s installs every 12 minutes.  Every 24 hours, AppLogic compresses those 12 minute “coin drops” and sends the compressed file back to the CA Technologies metering server.  Here’s the kicker – This has been happening for (GASP!) almost 7 years!  Since the inception of the product at a start-up called 3tera.

So what does one do with all this information?  Is it even valuable?  Well, one very smart guy, Bert Armijo, has been pushing for this data to continue to be collected for a reason. Initially so that the 3tera start-up could understand what services their customers were deploying on their technology and adjust their product strategy accordingly.  Then, overtime as more MSP’s wanted to be able to bill their customers for different types of services on different measurement metrics, the data set was expanded and the information was used to facilitate “Metered Billing” or “pay for what you use” cloud services.

Until recently, only a few of our MSP partners were using this data to bill their customers.  This few built elaborate systems to eat the mega compressed, flat files that AppLogic served up and punch out meaningful billing information.  And then there was me, and my colleagues attempting to figure out how many customers of ours were on the most recent version of our product or whether or not we should support older operating systems or browsers.  When we had a question like this, we’d go back to those lovely flat files, pull them together for a specific customer and repeat hundreds of times over to get the information we desired.  Then when the executives said something like “Oh, this is wonderful.  Can you add another field for my presentation tomorrow?” we knew our data was not only useful and valuable, but it was giving us a HUGE Big Data headache.  We had a volume, velocity and variety problem like no other and the data wasn’t even in a database yet!  Oh, and did I mention that those lovely flat files were compressed in some cases 8 layers?

So what did we do you ask? Well, we grabbed one of our well known data guys, a UI guy and a product manager (me) and we went to work creating a tool to be able to gather the compressed flat files and show basic reports from those files.  At first it was to get over our Big Data hangover but then… as we starting seeing the information come forth, we realized this information was a goldmine for our customers!

Not only would our customers be able to see what they have running, understand their capacity, manage their AppLogic licenses, or bill their customers, but they could use this metering data to boost revenue too.  Bert Armijo captured a few of these very valuable points in his MSP Mentor guest blog entitled “Mining Metering Data to Boost Revenue” back in February of this year.

Fast forward a few months from Bert’s blog post, we re-dedicated the team to creating a portal into this information that our customers could access.  It’s up and running today in beta.  One thing is for certain, every time we look at the information and fields being gathered, we can think up hundreds of reports we want to write, trends we want to track, statistics we want to pull.  Our executives want more and more each day – information is key, data is the new gold.  Or as Debra Danielson put it at the CA Exchange event “Data is the new oil“.

So back to the event and my attendance in the Big Data sessions.  I found out there that I’m not alone in this dilemma and that we have a team of experts called Data Scientists that can help us find the hidden trends in our data or determine things we can predict from the information we collect.  I’m excited for the prospect of people (other than our team of 3) looking at this information to help our customers gain more value from their data.  Imagine if we could predict when a customer would need more capacity or new services and get in front of that with a phone call/touch point.  Customer engagement to the next level and we’re not even retail shops predicting the next thing they should show  you on the shelves based on your browsing habits.

My little Big Data dilemma is not as rare as I thought.  Though we are not gathering exabytes of data each hour, we have a lot of data with a lack of capacity to consume.  But once we dug in, we found value right at the surface.  This is why I agree with Bernard Marr’s blog “Why the ‘Big Data’ Hype is NOT About Big or Data!

Big Data is not about how big the data set.  It’s about what you can get from the information that you never thought of before.  It’s about how much more educated you can be in managing your business and understanding trends that can effect your bottom line.

My advice to software product managers and engineers:  Find a way to gather data now.  It doesn’t need to have a complete architecture, or standardized on the latest platform. Instead of focusing on the tools, focus on getting the information and making it meaningful for you and your customers.

Super-storm Sandy: A Blessing in Disguise?

Our house during Super Storm Sandy and then after rebuilding

Our house during Super Storm Sandy and then after rebuilding

This past weekend, 11 months and 2 weeks from the day that SuperStorm Sandy devastated our area and destroyed our home, we moved back into our newly renovated and updated house.  It’s gorgeous, if I do say so myself.  The design is updated, everything is NEW and fresh and clean.  Everything is top notch or at least as top notch as we can afford.  We have a new foundation, walls, plumbing, electric, natural gas, paint, design, colors, furniture, bathrooms, kitchen, bedrooms, appliances, etc, etc… it’s truly amazing all that our contractor (Billy Flanzer of Flanzer Construction) and my husband have done to get this place in order.  I will say this:  I am blessed to be surrounded by all this beautiful stuff and to now have a home to enjoy with our family for years to come.

I’m so happy about moving back home that I almost forgot the heartache, stress and physical pain we all endured this past year.  My sense of optimism WANTS me to claim that the storm was a blessing in disguise.  Matter of fact, so many people have said it.  They say, “WOW, you made out after the storm”, “Oh how beautiful your home is, wasn’t this a blessing in disguise”, “We can’t feel sorry for you anymore! Your house is amazing!” or less directly “aren’t you glad you did all the work now?”

However, this pain and suffering is not yet over.  Despite the beautiful home, there are lasting negative effects of Super-storm Sandy that I don’t think I can forget just yet. (Sorry optimists!)  Here’s a few I simply can’t get out of my head…

1.  I have lost my sense of security that if you do the right things, you will be rewarded:  Thanks to the federal government, insurance and mortgage industries my sense of security and trust has been completely destroyed.  As I continue to fight for my insurance settlement, and get lost in unnecessary paperwork, phone calls, incorrect information, etc., they destroy me more and more each day.  This will not go away.  None of their processes, systems or people will change.  It saddens me (and worries me) that other people will have to deal with the same things we’ve dealt with this past year.  I really need to write a book or something to help these poor people!  Unfortunately, since the process and systems only get worse each day, and the people are less and less focused on helping others get through the mess, anything I write will be irrelevant in minutes.

2. My aching back, his cough, the kids’ unhealthy attachment to their toys… they’re all after effects of the storm.  I am physically injured due to stress and falling down the stairs, and then again falling through our broken deck and lifting heavy boxes through the 4 moves we made this past year.  My husband has been coughing since October 30, 2012… the day after the storm.  The mold, dust, exhaustion has gotten to him but unfortunately there is no test to prove it, no “cure” to take.  The only indication is his cough.  It’s constant and almost reflexive.  He doesn’t go 3 minutes without coughing and he’s a 30-something, healthy man.  My kids are confused, they don’t understand nor are as resilient as everyone says they are.  The baby (2 year old) hasn’t lived in one place long enough to know what is home.  She has anxiety when new people are around, she holds onto her “stuff” and is possessive over people taking her toys.  She has to bring EVERYTHING with her when we leave the house.  Now, this may be regular 2 year old behavior… but her night terrors are real, her confusion when she gets out of bed each night searching for mommy and daddy is real.  The 6 year old is more resilient.  She was just so happy to ride her bike again and have her room with all her things.  She’s excited to ride the bus to school again with her friends and have a play date with the kids in the area.  She’s planning fun things but has sacrificed a lot for a little kid. She’s also possessive over little things that she’s collected. It’s almost hording behavior and she won’t give up a thing no matter what compromise we make.

3.  Our financial situation is not pretty.  We’ve always lived within our means.  Both my husband and I work full time (60-70 hours a week) on salary, the kids are in school/daycare/aftercare, we take one vacation a year, focus on saving for retirement, pay our credit cards off each month in full, are never behind on the bills and don’t have to borrow from anyone.  In this past year all those ideals have broken down.  Our life savings is spent, we’ve borrowed A LOT of money, our credit cards are maxed and we’re only making minimum payments.  Yes we invested in our home to make it better than it was before but that all came at a huge cost.  We no longer have our emergency fund and we owe money to so many people that it’s a full time job to track.  I know we’ll get things back in order, but that comes at a cost of my own sanity and our credit score.

Well then, lets get off of this pity party.  We’re “blessed” in that we do have a semi-healthy family, a home to grow in (that is beautiful!) and a lifetime to live.  It was amazing eating our first home cooked meal.  Doing the basic stuff like playing with the kids on the floor or going for a walk down the block make us feel amazing.  I missed this so much.  I really felt my stress level go down the minute we moved back home.

Over the next few months I’ll continue to fight for our money from Wells Fargo, our mortgage company (they’re holding it in escrow), from our insurance company National Flood Insurance Program and from FEMA. We hope we will get grant money from the NY Rising program despite the fact that we make an income.  It took me 3 full days to fill out that application! AND last but not least, we will recover.

Our Family has only just started recovering – our mental recovery will take a while but being home was the biggest and first step in that recovery.  I only hope that others who are still rebuilding, still displaced and still begging for their money from their insurance and mortgage companies get what they need to move forward.

So, thank you to all our friends, family and neighbors who have wished us well along the way.  We are blessed to have you to help us through and hold us up through the difficult times.  I do not ask for your pity, only your support and understanding that going through a disaster like Superstorm Sandy will never really be a “Blessing in Disguise”.  I would much rather have renovated my house within my own means, on a schedule without uprooting my family and financial situation.

No, Sandy was not a Blessing.  But we learned from the experience and are stronger because of it.

-S

%d bloggers like this: