In this post I'll walk through extracting data from a database of one million contemporary popular music tracks, loading them into AWS Redshift via S3 and running some simple analysis via PostgreSQL's interactive terminal psql.
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks. The dataset is the result of a collaboration between The Echo Nest and LabROSA at Columbia University, supported in part by the NSF.
Within this dataset there are several subsets of data. The one I'm interested in is the million track metadata extract in SQLite3 format. The following will download it from Columbia University's Servers.
Before I load the data into Redshift I'll first explore the shape of the data and value distributions.
The database has a single table within it. Most of the columns aren't very granular in terms of data types so I'll need to explore how small I can make each column when designing the Redshift table schema later on.
I'll then check the cardinatily of the track and song identifier columns. The track_id column is unique where a song_id can have multiple track_ids.
I've created a query to pull six fields of interest from every record the table and I'll export that data out into a CSV file.
The fastest way to load data into Redshift is by breaking up your data into separate files and compressing them. Below I'll break the one-million-row CSV file up into twenty CSV files of 50K rows each and compress them.
You want to make sure your S3 bucket is created in the same region that you'll be launching your Redshift instance in. Amazon's us-east-1 region has the cheapest price on Redshift clusters so I'll use that region to create my S3 bucket in.
The cheapest Redshift cluster I could find on Amazon's pricing page was a dc1.large in us-east-1 at $0.25 / hour. It comes with 2 vCPUs, 15 GB of RAM, 0.16 TB of SSD-backed storage and throughput support of 0.20GB / second.
Once you execute that command Amazon will set about creating the cluster. In my experience it usually takes a few minutes. I tend to run a watch command to keep an eye on the cluster status and wait till I can see the cluster's endpoint address and see the status set to "available".
In the above you can see the security group is set to my default one. Within that group I've allowed my local IP address to connect on port 5439. I tend to set my security groups up via the AWS console rather than the CLI tool so I don't have a CLI example to copy-and-paste.
Once you've enabled access from your IP address to the Redshift cluster via port 5439 you'll be able to access it via the PostgreSQL interactive terminal. Below I'm using version 9.4.5. I've seen 9.3 connect to Redshift just fine as well.
I'll then issue a command to load in the data from the GZIP'ed CSV files on S3. This command has place holders where your AWS access key identifier and secret access key need to go. Please change these before executing this command.
When I ran this command 77MB of data (when uncompressed) was loaded in 9.92 seconds (~7.76 MB / second). The networking overhead of fetching so many small files probably contributed to the slow performance of this operation.
When I created the table in Redshift I took a guess as to the encoding methods to use on each field. With the data now loaded I can analyse 100K records and see which encoding method will result in the smallest amount of space being used. Redshift is primarily I/O constrained so generally the less disk space you data takes up the faster your queries will run. There are exceptions to this rule in relation to sort keys though.
Governments and Social Media companies are in the midst of a heated debate on how to regulate social media platforms. This can often fall into finger-pointing and mutual suspicion. For example, many Governments believe that social media companies like Facebook, Twitter and YouTube cannot be trusted to act in the public interest because they will always prioritise business interests. In my previous article “Policy Issues Facing Social Media Companies: The Case Study of YouTube”, I argued that social media companies are often not trading-off public interests for business interests. They are more often trading-off competing public interests, which creates many dilemmas that Governments may not understand.
This article goes a step further and argues that Governments must fundamentally shift their paradigm towards regulating social media companies, recognizing that social media companies, like Governments, are representations of public interests. Here it goes:
By enabling anyone to produce and share content, social media platforms like Facebook and YouTube have decentralized how information and opinions are shared in society. This has brought tremendous public value, such as freedom of speech and enabling access to education. However, it has also enabled individuals to spread hate speech, terrorist agendas and fake content, which can threaten national security and social harmony.
Some argue that the social media space should be completely free and left to the discretion of users. Users will rise up to counter offensive or fake material, or judge for themselves that these should be ignored.
This anti-regulation approach is irresponsible towards public interests. Targeted defamations and incitements to racist violence can easily go viral on social media platforms. Without swift actions by authorities, consequences to personal wellbeing and national security could be irreparable.
Some regulation is necessary to strike the balance between advancing free speech and protecting public interests such as national security and social harmony – the question is how.
I propose a new paradigm for how Governments regulate social media companies, which I term ‘Co-regulation’.
In the media space, Governments have traditionally seen themselves as guardians of public interest, enacting regulation to prevent content which violates standards of public decency. Governments must recognize that unlike traditional media companies, where content is generated by small group of individuals, social media platforms represent a broad base of content producers and users. Social media platforms, like Governments, are avenues for public interests to be represented.
Hence, Governments cannot see themselves as enforcers of public interest against social media companies. Instead, Governments and social media companies are joint stewards of public interests on social media platforms. This is the paradigm which undergirds ‘Co-regulation’.
First, content standards should be interpreted and operationalized on social media platforms through an inclusive mechanism. When it comes to interpreting content laws, the scale and speed of the digital world make court decisions impractical. While it would be expedient to assign responsibility to social media companies to interpret and operationalize content laws, this would be unrepresentative of public interests. One idea is for Governments and social media companies to co-develop a swift mechanism which allows a spectrum of public voices to influence the interpretation of content laws in grey cases.
Second, Governments and social media companies should establish a system of public accountability. A good example is the Code of Conduct on Countering Illegal Online Hate Speech, established by the European Commission and four major social media platforms in 2016. It sets public goals for how quickly illegal hate speech should be reviewed and removed. Results are published on a regular basis.
Third, Governments and social media companies should both make commitments, and be held jointly accountable, to public goals. For example, while social media companies invest in systems to detect and review potentially illegal content, Governments should engage the public on what constitutes ‘hate speech’ and ‘fake news’, so that user-flagging is more effective.
By implementing a law which enables hefty fines for social media companies which fail to take down ‘obviously illegal content’, Germany has argued that without legislation, social media companies will not take their responsibilities seriously.
In my view, the costs of legislation generally outweigh the benefits. The upside – better enforcement – is limited. Business incentives to remove objectionable content are already in play: advertisers are social media platforms’ main source of revenue, and none want their ads to be associated with objectionable content. An advertisers’ boycott on YouTube earlier this year suggests that market forces are alive and well.
On the other hand, legislation can have dangerous effects. Placing legal responsibility on social media companies to identify the lawfulness of content on their platforms creates an incentive to err on the side of greater caution, i.e. more censorship. Beyond undermining the right to free speech, companies may inadvertently censor important public feedback, for example, on Governmental corruption. Besides, enacting legislation sends a signal that social media companies cannot be trusted to act in the public interest, which is inimical to the principles of co-regulation.
Governments worldwide should recognise social media platforms as legitimate representations of public interests. As co-stewards of public interest, Governments and social media companies hold joint responsibility and accountability for regulating the social media space in a way that best represents public interests. It is about time Governments and Social Media Companies work collaboratively under this new paradigm of co-regulation.
<Just like all the articles on http://www.techandpublicgood.com, this article represents my personal views and not the view of my organization>
One of the goals of www.techandpublicgood.com is to bridge the worlds of Government, tech and business, which often hold a degree of suspicion towards each other. This article dives deep into controversial policy issues surrounding social media companies.
As a case study, it elucidates the challenges, considerations and dilemmas behind YouTube’s policies. This is me, a Government policy-maker, putting myself in the shoes of a YouTube policy-maker. I figure our considerations are similar despite our different contexts. If you know better than me on any of these issues, feedback is much, much welcomed.
We live in an increasingly divided world. The forces driving these divisions, for example, rising income inequality, geopolitical, racial and religious tensions, were in play long before the advent of social media.
However, social media has provided a channel for divisions to widen. Lowering the barriers for individuals to share and ‘viral’ their knowledge and opinions has brought tremendous benefits, such as spreading education and freedom of speech. On the other hand, it has given greater voice and reach to malicious or ‘fake’ content. Algorithms designed to push us to what we will most likely click create an echo chamber, reinforcing our beliefs and biases.
When a flurry of social media companies took to the scene in the 2000s, their intention was to create platforms for people to find what they wanted – friends, funny videos, relevant information, roommates or hobbyist items. Very few would have imagined that their platforms would completely change how everyday folks conversed and debated, shared and consumed information.
Today, social media companies are adjusting to the new responsibilities that this influence entails. Here is an overview of the issues at stake.
It is important to recognize the role of social media in democratizing how information is generated, shared and consumed. At the same time, not everything is appropriate to be shared online. Social media platforms recognize that they must have a moral view on harmful content that should be taken down, for example, content which aims to instigate violence or harm to others.
However, censorship cannot be overused. Social media platforms cannot become arbiters of morality because many issues are subjective, and it is not the platform’s role to make a judgment on who is right: The same LGBT content can be affirming for some, but offensive for others. When is it fake news, or merely a different interpretation? Here’s a real dilemma: let’s say someone reports an outbreak of disease on Facebook. The Government requests to take down the report until their investigations are completed because it will incite unnecessary fear in their population. Is Facebook best placed to assess who is right?
In general, a social media platform’s policy must identify and take down of content that is inherently harmful, while catering to subjectivity by providing choice – to users, on the content they receive, and to advertisers, on the content their brands are associated with. It is an intricate balance to strike, requiring nuanced, consistent policy backed up by a strong and coherent detection, enforcement and appeals regime.
Another policy area surrounds copyright. Individuals sharing content online may inadvertently or intentionally infringe on others’ copyrights. On one level, better detection of copyright infringements is needed. YouTube invested $60m in a system called ContentID, which allows rights holders to give YouTube their content so that YouTube can identify where it is being used.
What to do about copyright infringements is another issue. Should they be taken down immediately, or should the platform provide choice to copyright owners? Paradigms have shifted over the years in recognition that copyright owners may have different preferences: to enforce a take down, seek royalties or take no action.
First, how can the platform generate advertising revenues and keep their user base engaged, while respecting different preferences for personal privacy? This typically pertains to the practice of combining personal information with search and click history to build up a profile of the user, which enables targeted advertising. Information is sometimes sold to third parties.
Second, what does it mean to give people true ‘choice’ when it comes to privacy? Many argue that long privacy agreements which do not give people a choice other than quit the app do not provide people a real choice in privacy.
Third, should individuals have the right to be forgotten online? The EU and Google have been in a lengthy court battle on the right of private citizens to make requests for search engines to delist incorrect, irrelevant or out of date information returned by an online search for their full name, not just in their country of residence but globally.
Children bring these policy issues into sharper focus based on notions of age-appropriateness, consent, manipulation and safety. Platforms like Facebook do not allow users below 13. YouTube introduced ‘Restricted Mode’ as well as YouTube Kids, which filter content more strictly than the regular platform.
Similarly, higher standards apply to children’s privacy. Should companies be allowed to build profiles on children, and potentially manipulate them at such a young age? Should people be allowed to remove posts they made or online information about them while they were children?
Safety for children is also a huge issue particularly on interactive platforms where children can be groomed by predators. Taking into account privacy considerations, how can we detect it before harm is inflicted, and what is the right course of action?
I have not scraped the bottom of the barrel on the range of policy issues that social media companies deal with, but the broad categories are in place. Now let’s get into specifics of how social media companies have answered these questions through policy, implementation and resource allocation.
To put some meat on this, here’s a quick case study of YouTube’s approach. There are at least four components:
Product differentiation is one way to cater to different appetites for content and privacy. In 2015, YouTube has launched ‘YouTube Kids’ which excludes violence, nudity, and vulgar language. It also provides higher privacy by default through features such as blocking children from posting content and viewing targeted ads, and enabling them to view content without having to sign up for an account. ‘YouTube Red’ offers advertisement-free viewing.
However, product differentiation has its limits because significant resources are required for customization. There is also a slippery slope to avoid: if YouTube rolled out “YouTube China” with far stricter content censorship, imagine the influx of country requests that would ensue!
Providing users choice in their settings is another way to cater to varying preferences within a given product. For example, advertisers on YouTube may have varying appetites for types of videos their advertisements are shown against. Enabling choice, rather than banning more videos, is key: earlier this year, YouTube introduced features that enabled advertisers to exclude specific sites and channels from all of their AdWords for Video and Google Display Network campaigns, and manage brand safety settings across all their campaigns with a push of a button.
Concerning privacy, users who do not want their personal data and search/click history to be linked can go to the activity controls section of their account page on Google, and untick the box marked “Include Chrome browsing history and activity from websites and apps that use Google services”. For particular searches, you can also use “incognito mode”, which ensures that Chrome will not save your browsing history, cookies and site data, or information entered in forms. These are ways to provide real choices in privacy.
A robust policy defines clear principles which determine when content should be taken down or excluded from monetization opportunities and Restricted Mode. Implementation policy then becomes critical. With the large volume of content coming online every minute, it is impossible for YouTube employees to monitor everything. YouTube has to rely on user flagging and machine learning to identify copyright infringements or offensive content.
However, algorithms cannot be 100% accurate and often cannot explain why decisions are made. A robust appeals and re-evaluation process with humans in the loop is needed to ensure the integrity of the policy. More importantly, the human touch is needed to positively engage content producers (who hate to be censored).
In my previous jobs, we often quipped: “policy is ops”. It is no point having a perfect policy if enforcement and implementation simply cannot support it. Policy teams need a constant feedback loop with implementation teams, to bridge the ideal with the possible.
Finally, robust policy is necessary, but insufficient for social media companies. Strategic communications and advocacy are an absolute must.
Public criticism of a company’s policies can negatively impact business. Boycotts and greater Government regulation are examples. YouTube is swimming against a common but simplistic narrative that tech companies are simply trading of public interests in privacy and security for business interests such as the growth of advertising revenue.
Misperceptions about policies can also have dangerous impacts. A few years ago, Israel’s Deputy Foreign Minister met with YouTube executives, raising the issue of Palestinians leveraging YouTube videos to incite violence against Israel. She later released a statement which inaccurately suggested that Google would collaborate with Israel to take down this content. Google refuted this, but the nuance could have already been lost with segments of the public. YouTube’s policy of neutrality must come across clearly, even as lobby groups try to drag it into their agendas.
The purpose of Strategic Communications is to create a wide circle of advocates around YouTube’s policy stance so that negative press and misperceptions are less likely to take off. Elements of Strategic Communications include:
Going beyond the ‘what’ of policy, to the ‘why’. It is important to illuminate the consistent principles behind YouTube’s policy stances, as well as the considerations and trade-offs entailed. Channels such as blog posts enable this, since mainstream media is unlikely to provide the level of nuance needed.
Building strategic relationships and advocates. This includes entering into conversations and debates with your most strident critics, and building alliances with third parties who advocate your views.
Strong internal communications. Since social media companies themselves are run by an aggregation of people with different beliefs, it is essential that employees do not feel disenfranchised by the company’s policy stance.
Providing an alternative narrative. In addition, an important point for YouTube to make is that more is at stake than taking down offensive video content. Ultimately, we are all fighting against greater divisiveness and polarization in society. Although some elements of YouTube exacerbate this, YouTube can also make a huge dent in bridging divides. Hence, I love what YouTube is doing with “Creators for Change”, a program that cultivates creators who aim to counter xenophobia, extremism and hate online. These creators are working on web series on controversial issues, as well as educational workshops for students. They are using the YouTube platform to close divides.
It is far too simplistic to say that companies only pursue business interests, leaving Governments to protect public interests. Every new product, including social media platforms, is a double-edged sword, with the potential to bring us closer to or further from where we want to be as a society.
Both Governments and Social Media companies are trying to push us towards the first scenario. However, Governments will tend to advocate for more conservative policies as their primary objective is to minimize downside on issues such as national security, privacy and Government legitimacy. On the other hand, private businesses are simultaneously managing downsides while pushing the boundaries on issues such as free speech and revenue generation models.
A natural tension between these two positions is healthy as we decide, as countries and global communities, where we collectively fall on issues. This is how democracy works, after all.
This article is the second part in a two-part series by Dr Johnathan Ng of Epibone, on personalized medicine and public good.
In my first article, I gave an overview of personalized medicine: personalized disease-modifying drugs, autologous cell therapies and stem cell therapies.
It is easy to be swept away by the promises of precision medicine. In his speech on the Precision Medicine Initiative, then-President Obama made uplifting points, “And that’s the promise of precision medicine – delivering the right treatments, at the right time, every time to the right person… we want to have a nation in which the accidents and circumstances of our birth aren’t determining our fate, and therefore born with a particular disease or a particular genetic makeup that makes us more vulnerable to something; that that’s not our destiny, that’s not our fate – that we can remake it.”
Indeed, the potential benefits are tremendous, but so are the risks: in the form of escalating medical bills, with unproven – or worse still – harmful treatments. In this article, I give Governments and Healthcare providers three areas to pay attention to when it comes to personalized medicine: Regulation, Healthcare Finance, and democratizing the benefits of personalized medicine.
As new personalized treatment modalities emerge, regulators are facing increasing pressure to green-light interventions, even if clinical benefits are not clear – to provide patients with a chance to live.
A watershed case was the US Food and Drug Administration (FDA)’s ruling against a scientific advisory panel, in favor of patient advocacy groups to approve Exondys 51 marketed by Sarepta for treating Duchenne Muscular Dystrophy. Despite a majority (7 to 6) of the experts citing inadequate convincing clinical evidence, the FDA director greenlit the approval of Exondys 51 due to a lack of clinical alternatives. Many commentators felt that this case set a precedent for the approval of personalized medicine products based on surrogate endpoints without clinical benefits.
There are also many grey areas when it comes to regulating clinical trials. As with any emerging technology, the benefits come at a risk, which people desperate for a cure may be willing to take. Due to the exploratory nature of trials involving new treatment modalities, patient safety is often left in the hands of researchers: a simple search on ClinicalTrials.gov shows nearly 6000 clinical studies involving stem cells, some of which have not been approved.
Some of these trials have resulted in debilitating consequences. For example, severe adverse effects, some resulting in death, turned the spotlight on Juno Therapeutics’ lead CAR-T cell therapy for treating adults with late stage acute lymphoblastic leukemia. The risks of stem cell therapy are also not well understood. Although treatment with autologous fat-derived stem cells has been used for various indications, a poorly administered trial recently led to permanent eye damage in three elderly patients with macular degeneration (damage to parts of the retina in one’s eye).
Without proper regulation and well-controlled clinical trials, the safety and efficacy of stem cell treatments cannot be determined.
Some researchers show more caution than others. In the first human trial that uses an induced pluripotent stem cells (iPSCs) derivative, investigators from Japan successfully treated macular degeneration by administering retinal pigment epithelial cells grown from the patients’ own stem cells. Yet, after identifying a few mutations in the second patient’s cells, the RIKEN group decided to suspend the trial in September 2015 before obtaining clearance from health authorities in Japan to resume in February 2017. Perhaps all scientists and clinicians would do well to hold themselves to a similar standard.
Regulatory bodies such as the FDA must continually engage and balance the needs of the scientific, patients, and clinical communities in meeting these new regulatory challenges – unfortunately, there are no easy answers.
With the flood of new interventions, another issue to consider is cost. If all interventions are fully reimbursed (i.e. paid for) by state and private payers, the healthcare system will soon become bankrupt. Yet, if no help is given, the cost to patients of living longer is bankruptcy. The American Society for Clinical Oncology (ASCO) wrote in a brief that a patient living with cancer is now three times more likely to file for bankruptcy than a healthy person.
Policymakers must strike a fine balance of curbing the rapid rise in healthcare spending without disincentivizing innovation and depriving patients of access to life saving treatments.
When weighing the clinical benefits of a new drug product with the cost, healthcare economists typically apply a measure called the incremental cost effectiveness ratio (ICER) which takes the difference in cost between the new drug and existing alternatives and divides it by the change in quality adjusted life years (QALY). The National Institute for Health and Clinical Excellence (NICE) of the U.K., for example, sets an ICER limit of £30,000 per QALY gained for new drugs including targeted therapies. Most policymakers in the U.S. generally apply an ICER limit of USD$50,000 per QALY gained.
Early evidence suggests that personalized medicine tests are generally cost effective, with 20% of them resulting in cost saving and more than half achieving ICER of less than $50,000 per QALY gained. However, measures of cost effectiveness apply a single threshold to a heterogeneous population. If reimbursement was based on this alone, some people would receive more healthcare than they would choose, and others less. As such, commentators have noted that “reimbursement mechanisms for targeted therapies are still very blunt in an era of personalized medicine”.
Policymakers must leverage data and work with other stakeholders to improve reimbursement policies, especially taking into consideration the underserved population. Yet, the onus does not belong to the policymakers alone. Drugmakers, payers and clinicians are very much involved in the determining how drugs are priced and reimbursed. Recently, there have been exhortations by clinicians for more value-based pricing whereby reimbursement is contingent upon patient outcomes. The focus on outcomes could ensure that personalized medicine realizes its full clinical value. To achieve that, drugmakers could enter risk-sharing agreements with payers for partial reimbursement prior to demonstrating clinical effectiveness.
Alternatively, clinicians can also exert pricing pressure on drugmakers indirectly. In 2012, researchers from Memorial Sloan Kettering Cancer Center (MSK) evaluated the drugs Zaltrap and Avastin for treating colorectal cancer. Although Zaltrap cost twice as much as Avastin, the MSK researchers found no differences in efficacy between the drugs. Consequently, MSK decided to not recommend Zaltrap to patients and this led the drug’s co-marketer Sanofi to drop the price.
Together, stakeholders can work to ensure that personalized medicine is conscionable and cost effective.
Finally, perhaps personalized medicine should be about more than just the diagnosis and the cure. Personalized medicine could go a long way towards disease prevention and mitigation by engaging the laymen and teaching them to monitor and manage their own health. In a recent trip with my parents to their dental appointment at a polyclinic in Singapore, I could not help but notice that the Health Promotion Board set up a booth that encouraged senior citizens to get screened for colorectal cancer. Participants were instructed to fill out their information, collect samples of their stool at home in the kits provided, and send the kits back for analyses. Though seemingly mundane, campaigns like this are probably the most effective way of bringing personalized medicine to the masses.
Low cost point-of-care diagnostics can also help to bridge the divide between first world medicine and third world need for solutions. After all, if the goal of personalized medicine is to understand and improve lives, esoteric treatments will hardly do a majority of the public any good.
Finally, it is a positive development that countries are thinking about how to democratize the benefits of personalized medicine. For example, the U.S. National Institute of Health is collecting data from underserved populations that are historically underrepresented in biomedical research, so that they too can benefit from personalized medicine.
We have already seen the good that personalized medicine can do. Yet, if we want the broader public to benefit from personalized medicine while minimizing both the financial and clinical risks to society and patients, there is still so much more that we must do. Stakeholders must continue working together to advance the personalization of medicine, not for fame or fortune, but for the greater good.
Thanks for reading this two-part series on personalized medicine and public good. When I started this blog, one objective was to use it as platform for issue-experts in technology fields to give us mini crash-courses, and to sketch out the implications for society. I am sure Johnathan will be more than happy to discuss these issues further. Let me know if you’d like to be connected!
Today, I’d like to introduce you to my friend Bert Kaufman, Head of Corporate and Regulatory Affairs at Zoox, one of the hottest self-driving car start-ups in the Valley. I’ve previously written about why tech companies need policy teams more than ever, and Bert’s work is at the forefront of this.
When I first met Bert, we had already heard of each other, and immediately hit it off. In addition to being extremely kind and generous, Bert is a killer combination of a big-picture, systems thinker – from his days in Washington – and an embodiment of the generous, action-oriented, and creative start-up culture in the Valley, where he currently works.
In this interview, I ask him about his transition from Washington to Silicon Valley, issues surrounding self-driving, what he wishes Government folks knew, and how else he thinks technology should be harnessed for public good.
This is my third profile piece on folks who work at the intersection of tech and public good, following Xinwei Ngiam and Kenneth Tay. Enjoy!
Tell us a little more about yourself. Why did you move to the Valley after almost 8 years in Washington?
I spent most of my time growing up on the East Coast and down South, so from a cultural standpoint, I always thought Washington was a great mix of north and south—“The Northern most Southern city.” From a professional standpoint, I am a lawyer who loves policy issues, so I gravitated towards Washington after law school. But what I discovered about myself over the past decade is that I really love building organizations and organizing initiatives around good ideas. And there is no better place in the world to build these things than the capital of innovation and entrepreneurship. My move out to the Bay Area was prompted by my fiancée who was in graduate school at Stanford, and because my role as an appointee in the Obama Administration was winding down.
What did you enjoy most about your job in Washington and how did it prepare you for your current role at an autonomous vehicle startup?
Before joining the Obama Administration in 2013, I spent five years growing an organization called Business Forward. We started Business Forward to help business leaders from across the country do a better job of advising Washington policymakers and, conversely, to make it more efficient and transparent for policymakers to listen to business leaders. Through that experience, I faced the challenges of building an organization from scratch and learned the importance of taking a long-term view. Our funding came from about 60 large companies—our members—and I traveled around the country, engaged with thousands of people, and learned about issues that businesses of all shapes, sizes and ages faced as the country emerged out of the 2008-09 financial crisis.
That experience prepared me for the chance to join Penny Pritzker’s team at the Commerce Department. Not only did I get to work for an incredibly brilliant, demanding, and hard-working leader in Secretary Pritzker, but I also had the opportunity to help build and manage an initiative we created called the Presidential Ambassadors for Global Entrepreneurship. This initiative worked across The White House, Commerce and State Departments, USAID, the Small Business Administration, NASA, and with some of America’s most successful entrepreneurs to mentor, motivate, and in some cases fund aspiring entrepreneurs from across the U.S. and around the world. I also worked on policy issues related to the digital economy on areas like data privacy and cybersecurity.
In my role now, as an in-house lawyer working on policy in a sea of engineers and computer scientists, it’s important to communicate clearly and to understand the policy implications of the technology that we’re developing. This is important both within my organization and externally. Transportation is a highly-regulated space, for many important reasons. As a society, we want people to move around freely, but we also want to ensure that they can do so safely. The advent of autonomous vehicles will lead to innovation in road safety. What we are doing is so new that we have the opportunity to create best practices that can set the bar for future policy.
Policy is really important to any technology business intersecting with regulated markets. Technology startups that fail to consider policy or regulatory implications do so at their own peril. Conversely, regulators need to understand that the regulations should be nimble, flexible, and fair and not cumbersome. These principles will allow technology to advance on a level playing field.
What is one thing you’ve learned or experienced that you wish your colleagues in Washington had a chance to?
Meaningful innovation is hard and takes time, so it is important to take a long view. Government can and should catalyze and support innovation through funding basic and applied research and challenge grants. Government should set ambitious policy goals while at the same time leaving innovation to the private sector.
For example, between 2004 and 2007, DARPA (the Defense Advanced Research Projects Agency) set out some “Moonshot-like” challenges and put forth a modest amount of prize money for autonomous vehicle-related technology. Today, the payoffs are huge. The teams that competed in those challenges are the fathers and mothers of all the autonomous driving R&D now taking place across the entire automotive industry. In other words, a series of small government challenges have generated an enormous amount of private sector investment and job creation. Two lessons here: the first is that a little can go a very long way; the second is that government set a goal, got out of the way, and let academia and the private sector drive the evolution of the space.
Self Driving Car technology is one of the hottest areas in the Valley. What are a few things the international community should know about Self Driving Cars?
First, autonomous technology will usher in a paradigm shift as large as when we transitioned from the age of the horse and carriage to the age of the automobile. Getting around will allow for increased productivity, and for people who live in areas with poor access to public transit, it could make it easier to access jobs and opportunities. We will also think about real estate differently. For example, much of real estate today is built for and around the automobile. Think parking lots and parking garages. In a world of shared autonomous vehicles, demand for parking decreases.
Second, the first rollouts will happen in cities in a ridesharing model, not in vehicles sold to end customers. Cities can benefit from shared electric, autonomous transportation because it will ease congestion and decrease pollution. As more people move into cities, the idea of individual car ownership becomes less tenable. In this model, liability shifts away form individuals towards fleet managers and manufacturers.
Finally, and most importantly, safety is paramount. In the U.S., more than 35,000 people are killed every year in automobile collisions. Most of those fatalities are caused by human choice or error. Autonomous vehicle systems will be designed to interact safely on the roads with other road users like human drivers, pedestrians, and cyclists.
Moving away from Self Driving Cars, what is one problem in society today – perhaps one you encountered at the Department of Commerce – that you think we can solve more aggressively using technology?
I think that technology, correctly harnessed and understood, has the potential to improve the lives of many. Technology underpins most of our economy today, and it’s only going to compound over time, so we need to use technology to do a better job of training and educating people for the jobs of the future. Governments can identify important priorities and strategies and incentivize education and training so that people are prepared and trained for an evolving economy.
Finally, as I said earlier, if the private sector’s job is to drive innovation, government should work to ensure that there is an adequate social safety net in place for all people as the economy changes.
Uber recently made headlines with this feature by the New York Times, on how it uses behavioural insights to get drivers to work longer hours, and go to areas where there is high passenger demand. As Uber drivers are not employees, Uber has very little formal influence over their behavior – they can’t mandate how much drivers drive, or what area they cover. Behavioural nudges are a relatively costless way of getting drivers to do what Uber wants, instead of using monetary incentives. But is Uber particularly guilty?
The use of behavioural nudges to shape customer, constituent and employee behavior is certainly not unique. Examples abound, including:
Businesses and customers. Positive reinforcement is the bedrock of modern advertising. Jeff Bezos from Amazon famously said “through our Selling Coach program, we generate a steady stream of automated machine-learned ‘nudges’ (more than 70 million in a typical week).” Games like candy crush turn us into addicts by providing mini rewards in our brains, releasing the neurochemical dopamine and tapping into the same neuro-circuitry involved in addiction.
Governments and constituents. The UK Government has its own Behavioral Insights team, which has helped sign up an extra 100,000 organ donors a year and doubled the number of army applicants by simply changing default options and how emails were written.
Employers and employees. One of the reason Google will fix your car, take care of your health and food needs all in one place is because they know it will get you to stay longer and work harder.
Now that you’ve seen some examples, what exactly are “behavioral nudges”? Definitions vary, but I’ll boil it down to two things:
Applying an insight about a person’s decision-making calculus (that the person might not even know about himself!) to get him to make the decision you want.
The person is likely unaware that this tool is being used (unlike a law or a policy, which he actively shapes his behavior to comply with).
For example, the principle of “loss aversion” suggests that humans are more likely to respond to a potential loss, than a potential gain. When you want drivers to drive for two more hours, tell them they’d lose out on $200 if they didn’t. Don’t tell them they’ll gain $200. We are also a lot more vulnerable to peer pressure than we think. The UK Government managed to nudge forward the payment of £30m a year in income tax by introducing new reminder letters that informed recipients that most of their neighbours had already paid. Never underestimate the power of inertia – which is why companies adopt “opt-out” rather than “opt-in” clauses.
An in-depth article on leading thinkers in the field of behavioural science (Kahneman, Tversky, Thaler, Lewis), can be found here.
The potential for behavioural nudges is increasing with the proliferation of data about individuals. The more you understand how people make decisions – to work longer hours, to buy your product, to pay their taxes, to brush their teeth, to play a game, the more effectively you can nudge them towards your desired behaviour. Facebook knows more about me than I do. Uber knows more about their drivers than drivers themselves.
As a result, these companies can push buttons I didn’t even know existed. They have the potential to hack my operating system and change my behaviour.
Hence the ethical question of when a “nudge” becomes outright manipulation is more pertinent than ever.
Here are several ways to think about whether a “nudge” is being used ethically. <By the way, some people argue that it’s never OK to curb someone’s “moral freedom” through nudges, but I find that too idealistic – nudges have been used for time immemorial. It has to be a matter of degree.>
On the positive extreme, behaviours such as showing up at a doctors’ appointment, attending school or paying bills on time can be seen as actions that are positive for the individual. On the negative extreme, you could have outcomes such as an alcoholic purchasing more alcohol, or a suicidal person being nudged off the ledge.
There is huge scope for debate in between the extremes. Uber could argue that getting drivers to work longer hours during peak period is good for their earnings. Facebook would argue that repeatedly pushing advertisements that users are more likely to click helps them find what they need and like faster.
What is the distribution of benefits accruing to Uber vs the driver if the driver changes his behaviour? In this case, there seems to be a direct trade-off between Uber and drivers’ interests. As more drivers come onto the platform as a result of the nudges, drivers don’t benefit from surge pricing. On the other hand, Uber gets the benefit of more rides and hence more earnings.
Is there an intention to deceive? The author suggests that some of Uber’s methods nudged drivers towards geographical areas on the pretext of a surge, but when drivers got there, they found there was none. Even if this was not the intention, the asymmetry of information is unfair to drivers. More transparency is needed, perhaps by providing drivers a live feed of surge rates in various areas, including when surge is dropping.
The ‘opt-out’ technique is one of the most commonly used “nudges”: always set your preferred option as the default, and count on human inertia (or ignorance) to keep people there. If you are a Netflix user, you’ve experienced this: once your episode ends, the next one comes on automatically in ten seconds. It is a nudge to keep you watching, but you can turn off this feature permanently. Google and Facebook will send you personalized ads, but you can opt-out and get those replaced by randomised advertisements instead.
If you are an Uber driver, you can also temporarily turn off the forward-dispatch feature, which dispatches a new ride to you before the current one ends (keeping you constantly driving, just as Netflix keeps you constantly watching). However, there is no permanent way to turn it off. It will keep popping back on when you take a new ride: you have to be constantly proactive about stopping it if you don’t want to overwork. Does the lack of a permanent opt-out feature make Uber more guilty? Perhaps. But I would like to find out more about the design considerations of both Uber and Lyft before giving a definitive view (hit me up if you have further insight!).
Generally, how proactively institutions educate their users/employees about the opt-out function matters, as does how easy it is to opt-out.
So is Uber particularly guilty? On the surface it seems to. But want to hear my real answer? I have no idea, simply because much of the nudging that institutions do today is invisible, making it impossible to compare. We – as users, employees, constituents – do not even know that it is happening, and there is no legal obligation to tell us.
Hence, rather than ask whether Uber is guiltier than other institutions which deploy “nudges”, I believe the more important question should be: is self-regulation by these institutions sufficient? If not, does anyone have the moral high-ground to arbitrate? Should there be a system where institutions report their use of “nudges” and hold each other accountable? Would love to hear your thoughts.
Usually this error code shows up on the screen whilst joining to friend’s squad and really disturbs players well. We provide two methods for solving this error code2-0X0000E000 in Tom Clancy’s Rainbow Six Siege