Using NodeXL as one of the graphing tools, the graph shows the connections among the Twitter users who recently tweeted the word bigdata when queried on February 27, 2012, scaled by numbers of followers (with outliers thresholded). Connections created when users reply, mention or follow one another. The green lines are “follows” relationships, blue lines are “reply” or “mentions” relationships.

Almost 50% of respondents stated that they are currently using Big Data to “understand their customers better”, whereas 32% stated they are using social media for “interaction and promotion purposes.” The survey also found:

– 13% did not consider Big Data a priority, so far as stating it was “not on the agenda”
– Over 50% state that flexible delivery platforms are a priority for the next 1-2 business years
– 19% of respondents have deployed digital marketing practices across the enterprise
– 4% used location-based software to target customer promotions

The study also found:

– 52% believe that organizational structures not designed to take advantage of either Big Data or social media priorities 
– 51% say that lack of technology infrastructure and IT systems are a significant challenge
– 43% and 31% are having difficulty in finding functional and IT talent, respectively 

Big Data and social media do not ave to be mutually exclusive. A number of businesses are beginning to integrate the two, using Big Data solutions to analyze business content based on their social media activity.

Information is changing too quickly (velocity), is to expansive (volume), and too diverse (variety) to make it possible for any one person to have a reasonable chance of remembering/recalling enough to be singularly useful in complex decision making. As such, Casnocha notes that to address this issue:

- You need to know your network.

- You need to who in your network knows what.

- You need to know how to ask questions that elicit helpful information.

In essence, the ability to solve problems and make effective decisions is proportional to the amount of knowledge in your network:

Accessible Knowledge (AK) is proportional to number of networks you have x people/network x question/person x knowledge/question.

A very powerful equation.

But, while I agree with these points, Casnocha does not give concrete examples on how to use modern technology/social tools to realize value from them. For example, LinkedIn is THE network tool used by professional to source a network. Right? FaceBook tends to be used for personal and some professional network sourcing, but does not provide any real relationship information (connections like in LinkedIn).

Casnocha also does not identify what tools can be used to identify who in your network knows what. In LinkedIn, people can use their deductive reasoning to learn a lot about what a person has done as a means of knowing what they know. Are there tools that can do this? But even if there are, it is almost impossible to infer what people are truly capable of answering through inductive reasoning, which is a more important capability in effective decision making. This is a real gap in current social platforms.

In terms of how to ask questions, this is always an issue. English is a very ambiguous language and its interpretation is highly dependent on the educational level of the sender and receiver. Couple that with the highly unstructured nature of written content in social media and you open yourself to a lot of variance in results of knowledge calculus.

To more effectively manage and leverage Accessible Knowledge, social tools like LinkeIn need to start mining not only the connections, but the relative knowledge as well. It is only then that the artificial social networks, that are pervasively replacing our real ones, will become useful enough to support the fine grained decision making necessary for today’s complex problems.

POST UPDATE:

My colleague noted that the AK lack consideration of critical thinking or maybe it was my analysis that lacked critical thinking. Hum. Oh well, let’s assume it was the former and that he is absolutely right. People today have lost this critical ability. I even find myself now and then forgetting to think before I act.

Extending and revising the original thought, here how I would interpret it:

AK = Kp(t) x Tct [number of networks you have x people/network x question/person x knowledge/question]

where:
Tct is the critical thinking transformation function – this is your ability to perform with what you know
Kp is an efficiency factor dependent on the person and a point in time

That is, the sum total knowledge of your networks is transformed through the critical thinking process, which is spatially dependent on the individual and temporally dependent on when (mode, season, day/night, etc.) the thinking takes place.

I have only one reply, “The scariest place I have ever been is THE darkest place on earth. It is the place of neither happiness or misery. Nor is it a place of neither rich or poor, or even right or wrong. This is a place where nothing exists but the absence of nothing. It is that place you find yourself in where you begin to realize just how much you don’t know you don’t know.”

Usually at about this point I see in their face the “oh man, I shouldn’t have ask” kind of look. You know the one I am talking about. It is that one that people give when they realize they want to be some place else, but need to stay and listen out of respect. That’s the look.

Being a nice guy I usually let them off the “listening hook” by saying they can alway stop by later to talk about this in more detail if they want. But, before they go, I add that, “my death opened a door to this dark place just once and that is a story worth telling.” After that their faces change from “I need to get out” to “I want to know more.” From which we talk about the philosophy of death’s role in helping us all better understand our true limits.

So, why all this pontification around death, not knowing what we don’t know, life, the universe, and everything else? Because we are on the technological brink of being capable of systematically exploring this deep dark space in ways that where never possible just a few years ago. We are capable of discovering new endless opportunities by just looking through all that data (FaceBook, Twitter, CRM, etc.) we take for granted everyday. This is the world of Big Data and these profound outcomes are why this new capability is important to us all.

Ok, enough philosophy, for now. Back to the practical, technical, and business aspect of big data

1. Google now studies the timing and location of search-engine queries to predict flu outbreaks and unemployment trends before official government statistics come out. Interesting.

2. Credit card companies routinely pore over vast quantities of census, financial and personal information to try to detect fraud and identify consumer purchasing trends. While not new, the big data approach is improving accuracy and precision, as well as speeding up prediction times.

3. Medical researchers sift through the health records of thousands of people to try to identify useful correlations between medical treatments and health outcomes. I wonder if the healthcare insurance industry is taking advantage of this?

4. Companies running social-networking websites conduct “data mining” studies on huge stores of personal information in attempts to identify subtle consumer preferences and craft better marketing strategies. This is a subset of the Target case study.

5. A new class of “geo-location” data is emerging that lets companies analyze mobile device data to make intriguing inferences about people’s lives and the economy. It turns out, for example, that the length of time that consumers are willing to travel to shopping malls—data gathered from tracking the location of people’s cell phones—is an excellent proxy for measuring consumer demand in the economy.

These applications do beg the question about privacy, “When does now-casting – search through massive amounts of data to predict individual behavior – violate personal privacy?”

Beth Carter also covered the session and noted these take aways: 

1. Social media is technology: “We argue the opposite,” said Hougland. Instead, “social media is biology and consumer behavior.” He cited the addictive qualities that we display when we check our e-mail and our texts: “Dopamine is being released in our brain, which is the same process an addict goes through. Without this, we feel lonely.” His advice, then, for marketers, is to relate to consumers at a really personal level.

 This is a key observation that biology and sociology are at the core of social media. 

2. Social media is a bubble: Hougland would argue that it is “absolutely” a bubble, and that social media is going to alter the marketing landscape for years. Media behavior doesn’t drive consumer behavior, it is reflective of consumer behavior.

Advertiser and marketers need to better appreciate that they can only add to a community, but not shape it. 

3. Brand matters more: Social media is actually largely dilutive of brand. When we share a brand to our friends, it’s our brand — people matters more than a corporate brand. Your brand is nine times more important than the opinion of a stranger. Brands see erosion if they don’t adhere to this. Brands needs to create a one-to-one relationship with everyone in the world.

 Again, this goes back to understanding the fundamental of influencing people through psychology and sociology.

4. The web site is the center of the customer journey: The customer journey is alinear, and we don’t know the zero moment. The world is social, and we must socialize every step of the customer journey, leaving no dead ends. The average attention span has shrunk in 10 years from 14 minutes to four, which should change everything we do in marketing. “Influence is Money Ball for marketers,” said Hougland. “We can measure campaigns in real-time through social data.”

With attention span exponentially decreasing, no single event will effectively shape an individual opinion. As such, one needs to think about the complete social value chain in terms of multiple points of contact.

5. Great creative drives disruption: Science and math are trumping art, and social media is the world’s biggest focus group — data will decide creative decisions. Your marketing organization may be digital but not social. The best model we found is to organize your brand like a social network, and the companies who use social data in real time for feedback are running better businesses.

This is so true, math and science are now trumping art, and is why big data is becoming the primary delivery means through which social media will be valued.

The quadrafication of big data (infrastructure, tools, exploratory methods, and people) is having a positive impact on these kinds of ecosystems. I believe we can use the big data reference architecture as the basis for refactoring the traditional catastrophic simulation, modeling, and financial analysis activities. Using platforms like Pneuron, we can help them more effectively map computationally complex MDMI (multi-data mult-instruction) workstreams into disaggregated process maps functioning in a MapReduce format, potentially using some of the existing simulation models. They could get the benefit of their a prior knowledge (models, tools), while dealing with the growth in data sets. Just a few thoughts.

One last note – this is an exercise in science and not engineering, or even systems integration. The practices that make for excellent enterprise architectures, requirements development, or even software engineering are of very little use here (those beyond critical thinking). To solve this problem, one must be willing to fail, fail early, an fail often. It is only through these failures that the true realization of Big Data Cat Modeling capabilities will be found.

Examination of the data flows in the evolving datasets shows trends and information without the need of prior working hypothesis, changing the traditional paradigm of social analysis in which the design of the study sample represents the first step which is than followed by the analysis of raw data. 

The key point is that big data is a catalyst for the exploration of level three knowledge: those things we don’t know we don’t know. Traditional discovery methods, used in level 1(know what we know) and 2 knowledge (know what we don’t know) knowledge acquisition, are limited because they infer, and often require, the relevant aspects of the questions under study. In contrast to the traditional deductive approach of knowledge acquisition, the big data is self-explanatory and can be based on inductive knowledge acquisition, which fundamentally requires large amounts of information. 

Therefore, if one believes that transformative business insights can be found in understanding more of what we don’t know we don’t know (level 3), then the use of big data is one of the proven means to achieve this.

Paul Butler, a FaceBook intern, discovered this earthly visualization using R, an open source statistical application. Based on around 10 million samples of Friends relationships taken from FaceBook’s Hadoop Apache Hive (data warehouse system for Hadoop), he was able to plot the weights for each pair of cities as a function of the Euclidean distance between them and the number of their respective friends. The result is this astonishing image of the earth. By the way, notice anything missing in the geographical location where China normally resides?

Here is another awesome image, this time created by Eric Fischer using a heat map (a type of visualization tool) of places with Flickr photographs and Tweets.  While Twitter and Flickr may not individually have enough users to create a map so detailed as the FaceBook image, but put them together, and there’s a wealth of information that can be discovered in all of its glory.

Data visualization is a powerful exploratory tool that should be exploited as early as possible when working to monetize your big data. Do not assume away discoveries just because you don’t know what you don’t know (third level of knowledge). Let the data take your brain on an unguided journey where discovery is the ultimate destination.

1. Data (the intrinsic 1/0 property of big data) which can be broken down subjective areas like interaction data, transaction data || structured, unstructured || realtime/streaming, batch/static || etc.

2. MapReduce platforms – AKA divide and conquer – virtual integration capabilities that enable aggregation and management of multiple name-spaced data sources (Hadoop, InfoSphere Streams, Pneuron, etc.)

3. Data Exploration, Data Mining, and Intelligence Platforms – technical capabilities that  enable one to derive insights from data (Pentaho, IBM InfoSphere, ListenLogic, MatLab, Mathematics, Statistica, etc.).

4. Knowledge Worker platform (AKA The human component) – The two most important capabilities come from data scientists (navigate through data) and  behavioral scientists (navigate through human behavior, which most important things seem to connect back to).

In essence, Big Data has data, an ability to find it and use it, and an ability to explore and learn from it.

Does this seem right?  Missing anything? Please post or email me.