Wednesday 30 September 2015

Data Mining for Predictive Social Network Analysis

http://www.toptal.com/
BY ELDER SANTOS - SOFTWARE ENGINEER @ TOPTAL
Social networks, in one form or another, have existed since people first began to interact. Indeed, put two or more people together and you have the foundation of a social network. It is therefore no surprise that, in today’s Internet-everywhere world, online social networks have become entirely ubiquitous.
Within this world of online social networks, a particularly fascinating phenomenon of the past decade has been the explosive growth of Twitter, often described as “the SMS of the Internet”. Launched in 2006, Twitter rapidly gained global popularity and has become one of the ten most visited websites in the world. As of May 2015, Twitter boasts 302 million active users who are collectively producing 500 million Tweets per day. And these numbers are continually growing.
Given this enormous volume of social media data, analysts have come to recognize Twitter as a virtual treasure trove of information for data mining, social network analysis, and information for sensing public opinion trends and groundswells of support for (or opposition to) various political and social initiatives. Twitter Trend Topics in particular are becoming increasingly recognized as a valuable proxy for measuring public opinion.
social network analysis and data mining
This article describes the techniques I employed for a proof-of-concept that effectively analyzed Twitter Trend Topics to predict, as a sample test case, regional voting patterns in the 2014 Brazilian presidential election.

The Election

General presidential elections were held in Brazil on October 5, 2014. No candidate received more than 50% of the vote, so a second runoff election was held on October 26th.
In the first round, Dilma Rousseff (Partido dos Trabalhadores) won 41.6% of the vote, ahead of Aécio Neves(Partido da Social Democracia Brasileira) with 33.6%, and Marina Silva (Partido Socialista Brasileiro) with 21.3%. Rousseff and Neves contested the runoff on October 26th with Rousseff being re-elected by a narrow margin, 51.6% to Neves’ 48.4%. The analysis in this article relates specifically to the October 26th runoff election.
Partido dos Trabalhadores (PT) is one of the biggest political parties in Brazil. It is the political party for the current and former presidents, Dilma Roussef and Luis Inacio Lula da Silva. Partido da Social Democracia Brasileira (PSDB) is the political party of the prior president Fernando Henrique Cardoso.

Data Mining and Extracting Twitter Trend Topic Data

I began social media data mining by extracting Twitter Trend Topic data for the 14 Brazilian cities for which data is supplied via the Twitter API, namely: Brasília, Belém, Belo Horizonte, Curitiba, Porto Alegre, Recife, Rio de Janeiro, Salvador, São Paulo, Campinas, Fortaleza, Goiânia, Manaus, and São Luis.
I queried the Twitter REST API to get the top 10 Twitter Trend Topics for these 14 cities in a 20 minute interval (limited by some restrictions that Twitter has on its API). Limiting the query to these 14 cities is done by specifying their Yahoo! GeoPlanet WOEIDs (Where On Earth IDs).
For this proof-of-concept, I used Python and a Twitter library (cleverly called “twitter”) to get all the social network data for the day of the runoff election (Oct 26th), as well as the two days prior (Oct 24th and 25th). For each day, I performed about 70 different queries to help identify the instant trend topics.
Below is an example of the JSON object returned in response to each query (this example was based on a query for data on October 26th at 12:40:00 AM, and only shows the data for Belo Horizonte).
[{"created_at": "2014-10-26T02:32:59Z",
  "trends":
 [{"url": "http://twitter.com/search?q=%23GolpeNoJN",
   "name": "#GolpeNoJN", "query": "%23GolpeNoJN", "promoted_content": null},
  {"url": "http://twitter.com/search?q=%23SomosTodosDilma",
   "name": "#SomosTodosDilma", "query": "%23SomosTodosDilma", "promoted_content": null},
  {"url": "http://twitter.com/search?q=%23EAecio45Confirma",
   "name": "#EAecio45Confirma", "query": "%23EAecio45Confirma", "promoted_content": null},
  {"url": "http://twitter.com/search?q=Uilson",
   "name": "Uilson", "query": "Uilson", "promoted_content": null},
  {"url": "http://twitter.com/search?q=%22Lucas+Silva%22",
   "name": "Lucas Silva", "query": "%22Lucas+Silva%22", "promoted_content": null}, 
  {"url": "http://twitter.com/search?q=%22Marcelo+Oliveira%22",
   "name": "Marcelo Oliveira", "query": "%22Marcelo+Oliveira%22", "promoted_content": null},
  {"url": "http://twitter.com/search?q=Cruzeiro",
   "name": "Cruzeiro", "query": "Cruzeiro", "promoted_content": null},
  {"url": "http://twitter.com/search?q=Tupi",
   "name": "Tupi", "query": "Tupi", "promoted_content": null},
  {"url": "http://twitter.com/search?q=%22Real+x+Bar%C3%A7a%22",
   "name": "Real x Bar\u00e7a", "query": "%22Real+x+Bar%C3%A7a%22", "promoted_content": null},
  {"url": "http://twitter.com/search?q=Wanessa",
   "name": "Wanessa", "query": "Wanessa", "promoted_content": null}
 ],
  "as_of": "2014-10-26T02:40:03Z",
  "locations": [{"name": "Belo Horizonte", "woeid": 455821}]
}]

Brief Intro to Social Network Analysis

Social Network Theory is the study of how people, organizations, or groups interact with others inside their network. There are three primary types of social networks:
  • Egocentric networks are connected with a single node or individual (e.g., you and all your friends and relatives).
  • Socio-centric networks are closed networks by default. Two commonly-used examples of this type of network are children in a classroom or workers inside an organization.
  • Open system networks are networks where the boundary lines are not clearly defined, which makes this type of network typically the most difficult to study. The type of socio-political network we are analyzing in this article is an example of an open system network.
Social networks are considered complex networks, since they display non-trivial topological features, with patterns of connection between their elements that are neither purely regular nor purely random.
Social network analysis examines the structure of relationships between social entities. These entities are often people, but may also be social groups, political organizations, financial networks, residents of a community, citizens of a country, and so on. The empirical study of networks has played a central role in social science, and many of the mathematical and statistical tools used for studying networks were first developed in sociology.

Establishing the Network

To create a network using the Twitter Trend Topics, I defined the following rules:
  • Each city is a vertex (i.e., node) in the network.
  • If there is at least one common trend topic between two cities, there is an edge (i.e., link) between those cities.
  • Each edge is weighted according to the number of trend topics in common between those two cities (i.e., the more trend topics two cities have in common, the heavier the weight that is attributed to the link between them).
For example, on October 26th, the cities of Fortaleza and Campinas had 11 trend topics in common, so the network for that day includes an edge between Fortaleza and Campinas with a weight of 11:
In addition, to aid the process of weighting the relationships between the cities, I also considered topics that were not related to the election itself (the premise being that cities that share other common priorities and interests may be more inclined to share the same political leanings).
Although the order of the trend topics could potentially have some significance to the analysis, for purposes of simplification of the proof-of-concept, I chose to ignore the ordering of the topics in the trend topic list.

Network Topology

Network topology is essentially the arrangement of the various elements (links, nodes, etc.) of a network. For the social network we are analyzing, the network topology does not change dramatically across the 3 days, since the nodes of the network (i.e., the 14 cities) remain fixed. However, differences can be detected in the weights of the links between the nodes, since the number of common trend topics between cities varies across the 3 days, as shown in the comparison below of the network topology on Day 24 vs. Day 25.

Predicting Election Results Using Twitter Trend Topic Data

To assist us in predicting election results, we consider not only the trend topics in common between cities, but also how the content of those topics relates to likely support for each of the two principal political parties; i.e., Partido dos Trabalhadores (PT) and Partido da Social Democracia Brasileira (PSDB).
First, I created a list of words and phrases perceived to indicate a positive leaning toward, or support for, one of the parties. (Populating this list is admittedly a highly complex task. In the context of this proof of concept, I deliberately took a simplified approach. If anything, this makes the caliber of the results all the more intriguing, since a more highly tuned list of terms and phrases would presumably further improve the accuracy of the results.)
Then, for each node, I count:
  • the number of its links which include terms that indicated support for PT
  • the number of its links which include terms that indicated support for PSDB
Using the city of Fortazela again as an example, I ended up with counts of:
Fortaleza['PT'] = 56
Fortaleza['PDSB'] = 37
We thereby draw the conclusion that Fortaleza residents have an overall preference for Partido dos Trabalhadores (PT).

Results and Conclusions

Based on this algorithm, the analysis yields results that are surprisingly similar to the actual election results, especially when one considers the general simplicity of our approach. Here’s a comparison of the predictive results based on the Twitter Trend Topic data as compared with the real election results (red is used to represent Partido dos Trabalhadores and blue is used to represent Partido da Social Democracia Brasileira):
social network analysis and data mining
Improved scientific rigor, as well as more sophisticated algorithms and metrics, would undoubtedly improve the results even further.
Here are a few metrics, for example, that could be used to infer a node’s importance or influence, which could in turn inform the type of predictive analysis described in this article:
  • Node centrality. Numerous node centrality measures exist that can be employed to help identify the most important or influential nodes in a network. Betweenness centrality, for example, considers a node highly important if it forms bridges between many other nodes. The eigenvalue centrality, on the other hand, based a node’s importance on the number of other highly important nodes that link to it.
  • Clustering coefficient. The clustering coefficient of a node measures the extent to which a node’s “neighbors” are connected to one other. This is another measure that can be relevant to evaluating a node’s presumed degree of influence on its neighboring nodes.
  • Degree centrality. Degree centrality is based on the number of links (i.e., connections) to a node. This is one of the simplest measures of a node’s “significance” within a network.
But even without that level of sophistication, the results achieved with this simple proof-of-concept provided a compelling demonstration of effective predictive analysis using Twitter Trend Topic data. There is clearly the potential to take social media data analysis even further in the future.

7 steps to avoid for mobile marketing automation heaven

pocketgamer.biz
Shannon Jessup is the senior vice president of partnerships for Tapjoy, a Personalized Monetization Platform whose mission is to help freemium mobile app publishers maximize their revenue.
The premise of mobile marketing automation is that it is supposed to help app developers do their jobs better, faster and more easily.
By enabling marketers and app developers to automate the delivery of targeted, relevant and timely messages to thousands or even millions of users on a one-to-one basis, the end goal of mobile marketing automation is to lift an app’s KPIs.
Unfortunately, it doesn’t always work out that way.
The tools and technologies that empower mobile marketing automation are only as good as the marketers that use them, and since the tools themselves are still relatively new to the market, there is a natural and expected learning curve in getting the best results from them.


No matter how reliable the technology itself is, marketers are going to make mistakes, and results aren’t always going to be as expected. When mistakes aren’t caught in time, marketing automation can even cause an app’s KPIs to move in the wrong direction.
The good news is, these mistakes can usually be corrected fairly easily. The first step is simply knowing what went wrong. To that end, we’ll provide the top mistakes many app developers make when carrying out a mobile marketing automation strategy.

1 Poorly defined user segments

In any engagement, retention or monetization campaign, the most effective results come when marketers are able to send the right message to the right user at the right time. When any one of those three variables is not tuned correctly, the whole campaign can fall apart.
So the first question that must be asked when an automated campaign fails to achieve its goals is whether it was being sent to the right segment of users in the first place.
In most cases, segments are too broadly defined. A good rule of thumb is to be as specific as possible when targeting to ensure that the vision of one-to-one marketing is as close to reality as 

2 Ineffective content or offers

Content is the key driver of successful marketing automation. The right content, delivered on time and on target, will capture your user’s attention and compel them to take action.
The wrong content, however, will not only fail to generate interest but could even damage your relationships if users start to feel as if you’re not listening to their needs or addressing their interests.
There are many forms your content could take, from IAP promotions designed to increase revenue to app tips designed to increase engagement. The key is to make sure the content is useful, interesting or actionable-or preferably a combination of all three.

3 Bad timing

Why do telemarketers always seem to call just when we sit down for dinner? No matter how relevant or interesting their pitch is, we are almost always irritated when that happens.
Mobile marketers must consider the timing of their messages as well - not only the time of day (especially for push notifications), but the timing of what is happening in the app at that moment.
Within just about every app there are certain moments that make the most sense to deliver a contextually relevant message, such as moments of failure, success, pause, opportunity and more. When marketers try to serve messages during inopportune times, they run the risk of disrupting and thereby annoying their users.

4 Using the wrong channel

As a mobile marketer, there are essentially three channels of communication at your disposal: push notifications, in-app messages, and email.
Each channel has its own strengths and befits certain types of messages depending on their objectives.
  • In-app messages can reach users in real-time as they actively engage with the app and are best used for contextually relevant engagement or monetization messages.
  • Push notifications are able to reach users outside of the app and can help engage lapsed customers or notify users of important app updates or events.
  • Email can be a valuable tool for reaching users who have not opted-in to push. If your message isn’t getting the right response, it might be getting sent through the wrong channel.

5 Being too pushy

Just because your goal is to get users to make an in-app payment (or monetize in some other way), it doesn’t mean that every marketing campaign should be geared towards an IAP.
With marketing automation, your main goal should be to nurture relationships with your users and drive engagement to the point where they actually want to make an in-app payment as opposed to being “forced” into it.
Sending messages that urge them to make an IAP at practically every turn will only wear them out and have the opposite affect of what you intended. But providing valuable messages that help improve their experience will increase retention and loyalty and ultimately stands a better chance of leading to a payment down the road.

6 Failure to analyze data

Analytics and automation go hand in hand. The insights gleaned from studying your user’s behaviors and in-app activities will dictate the types of marketing automation campaigns you deliver.
But too many marketers make the mistake of simply assuming they understand their users and know what they want, without actually using analytics to confirm their theories. This can have dangerous repercussions on user relationships, especially if the marketer starts sending users irrelevant or untargeted messages.
By tapping into app analytics and really getting to know your users, you’ll know what types of marketing campaigns are likely to resonate with them most.

7 Setting it and forgetting it

It’s easy to fall into the trap of thinking that just because something worked the first time, it will continue to work well into the future.
One of the worst mistakes you can make when managing a mobile marketing automation campaign is taking the “automation” part a little too far.
“Automation” doesn’t mean that you can simply set it and forget it. All marketing automation campaigns still require a certain level of manual oversight.
If your campaigns have been on autopilot for too long, it’s time to consider updating the copy, offer, message, targeting, timing or creative to keep the campaign fresh and make sure it’s working as effectively as it should be.
While mobile marketing automation can be a very powerful way to improve your user engagement, retention and monetization efforts, it is not a panacea for poor marketing.
There are no substitutes or shortcuts for proper planning. By avoiding the common pitfalls described above, you’ll be well on your way to getting the most from your mobile marketing automation campaigns

Android Pay, ISIS and the road to mobile payments ubiquity

mobilepaymentstoday.com
At the Mobile Payments Conferene in Chicago last month, Jack Connors, Google's head of commerce partnerships, told attendees that Android Pay would launch "very soon." A week later, Google announced a soft rollout of its revamped mobile wallet. 
Android Pay is 100 percent focused on the merchant relationship with their customer, Connors said at the conference. By positioning the merchant as the owner of the relationship, Android Pay will leapfrog the current offering from Apple and move the mobile payments industry to the next generation and closer to ubiquity among most consumers.

Making the leap forward

In order to understand Android Pay, we have to start with ISIS. Yes, the notorious terrorist group inadvertently has a hand in shaping the mobile payments landscape in the U.S. 
Prior to ISIS being a terrorist organization, Isis also was the name of a joint venture between AT&T, T-Mobile and Verizon created in 2010 that was intended to be a mobile payments platform, but evolved into a mobile wallet offering. Isis was forced to change its name, for obvious reasons, to Softcard and the brand never fully recovered. Google acquired Softcard and its IP in February 2013 and wasn't heard from much again until the Mobile Payments Conference where Connors described how Softcard's Smart Tap technology is at the center of Android Pay's value proposition for consumers and retailers.

Smart Tap

Smart Tap uses NFC to transfer not only payment information but also loyalty, rewards, coupons, discounts and offers wrapped in a secure tokenized transaction. The rewards programs, discounts, offers, points, etc. create a relationship between the consumer and the retailer, and these 'relationships sit inside "Google Wallet" and the Android Pay app facilitates the transaction as the retailers deepen their relationship with their customers.
And this makes a lot of sense. Retailers have the advantage over handset makers, mobile wallet providers and tech companies when it comes to building relationships with consumers. They know their customers, interact with their customers, they have the products their customers want, they own the stores the customers are shopping in, they have the money coming in from their customers and they have the ability to set the prices. The retailers should own the relationship not the wallet providers, handset makers or tech companies.
But isn't this what Google has been doing all along? Fostering relationships between retailers and consumers through its search engine and, in turn, creating value for consumers and retailers by connecting them through its various advertising and search platforms. Google is now using a similar "retail-centric" strategy with Andriod Pay at the point of sale. 
In a video of Android Pay in action, a consumer walks up to a Coke vending machine, ukulele music playing in the background, and hold their phone up to the NFC reader to purchase a Coke. Because the consumer has signed up and included their Coke rewards in their Andriod Pay app, the consumer's Coke Rewards details are displayed on their phone's screen. Once the transaction is completed through Smart Tap, Coke rewards are earned. If the consumer makes three more purchases, they receive a free Coke. 
The consumer didn't have to enter a password or remember or remember their rewards number to complete the purchase. It was an easy, secure transaction and rewards were earned by the consumer and the Coke brand has a more loyal customer. 

The road to ubiquity?

Connors said his expected timeline for mobile payments ubiquity is 18-to-24 months. Why is he so bullish on the future success of mobile payments given where we are today? 
  1. I do not think it is a coincidence that the launch of Android Pay coincides with the shift EMV, which has been speculated to negatively impact the checkout experience for consumers with consumers dipping instead of swiping their card and PIN requirement for credit cards. This could be an additional push for consumers to choose the easier, faster and equally secure payment path, which is mobile payments. Not to mention that mobile payments will come with increased offers, discounts and promotions in 2016.
  2. Connors said “I will be very surprised if, come November, an Android user doesn’t know about Android Pay,” which tells me that Google is serious about making Android Pay a success and that there will be significant media support and signage to build awareness in the next couple of months on air and in retail locations. This is in addition to the media spending by Apple, the issuing banks and the card operators, which will continue to lift the awareness, value proposition, ease, security and coolness of mobile payments in the consumers' minds. 
  3. The strong market share and sheer number of Android-enabled phones in the U.S. will help as well with Android at a 66 percent market share vs. Apple’s 30.1 percent. There are also some 1,000 smartphones that run on the Android operating system, according to MacDailyNews. This is a huge advantage over Apple Pay but it also enables a broad spectrum of millions of consumers to make mobile payments. This has never happened before. 

The value proposition of seamless and secure mobile rewards, coupons, offers and discounts at the point of sale, the enormous marketing machine that is Google, and the shift to EMV cards and the market share of Android-enabled phones will combine to position Android Pay as a dominant player in mobile payments. Two thousand sixteen will not only be an interesting year for politics, but it will also be an incredible year for mobile payments. 

Tuesday 29 September 2015

Mobile Marketing, Location, and Attribution

huffingtonpost.com
Resultado de imagen para mobile marketing
By Mark Campos | Product Expert, Waze
Smart marketers will embrace mobile. They'll deliver useful, relevant brand messages with context, driving more customers to store, more eyeballs on the season premiere, or increase purchase of their new menu items. Their ads won't look anything like right-rails or mobile banners, and they won't be measured by CTR. Consumer attention, followed by budgets, are shifting toward mobile, so it's time to consider how the medium should be used, what makes it unique, and how we can prove it's effective.
Clumsiness in New Media
Early television advertising was essentially graphic radio advertising--voice reading over moving text. In the 90s, web marketing was print marketing redux. New mediums often begin with pasting the old into the new, expecting to measure and drive the same kinds of engagement. We see the same on mobile today, and as marketers, only have ourselves to blame. Mobile banners are a dull use of the medium, and yet they constitute 82% of mobile advertisingWe know that mobile banners are widely ignored (with 38% of clicks being accidental), and it's quickly becoming acknowledged that their days are numbered. To be smart, we have to understand context.
Context Matters in Mobile
Our phones are the only screen we carry from the bed to the bathroom to the boardroom. This screen is beside us more often than not, whether we're watching TV, shopping for groceries, running through the park--the list is endless by now. Therefore it's counterintuitive that the data we use to target and measure activity on mobile are so blunt. These devices are packed full of sensors, they're recording great data, why aren't we able to use all of it? Firstly, users are in app silos. Of all time on mobile, 86% is spent in-app. We can still reach users in native apps, but when marketers buy that inventory blindly, they lose context. Critical user signals are lost in mobile ad networks; marketers must find signals that are common denominators across hundreds or thousands of apps, which ultimately compromises their understanding of context and effectiveness. With such a vague understanding of user behavior at the moment of an ad impression, we're unable to deliver ads of value or measure their effectiveness. Native ads today deliver 18% higher lift in purchase intent, over networks. And if there's one signal that mobile marketing should always consider, its location.
Location Marketing's Dirty Little Secret
Location signals today in mobile marketing are undeniably valuable yet woefully misunderstood. AdExchanger sums up the irony of location inaccuracy through deep research, concluding that, "Less than 1% of location data from ad exchanges is accurate enough to help marketers understand people's movements in the real world". Even with precise signals, real value isn't found in reaching people simply because they're nearby. There are marginal performance lifts in geo-awareness and geo-fenced campaigns, but again, context is king. Native apps, by contrast, are a unique channel for marketing, with strong signals of user location, behavior, context, and goals. The mobile device is persistent throughout our daily routines, as marketers we should be able to see and take advantage of that data. Native location apps give us rich behavioral profiling, valuable context, and an audience that sees and responds to ads.
Measuring Effectiveness
There is no one-size-fits-all for measuring mobile marketing at scale. There are panel-based solutions, which only measure a portion of your audience, and may have conflicts of interest. There are third party solutions, like connecting data through transactions at the register, but require expensive data subscriptions and complex technical integrations. On a native ad platform, you aren't going to see the scale of impressions, clicks, and purchases you'll see with other solutions, but the results you do see will be authentic. Marketers are learning the only end-to-end solutions that can measure ad effectiveness from impression to purchase are first-party, native platforms with accurate location signals.