Roughly 4.4 billion, or 56% of the world’s population, live in cities, making them some of the most interesting, exciting, and dynamic places to be. At the same time, coordinating services for millions of people can be a constant challenge for policymakers and public institutions.
Today’s global cities require extensive connectivity to optimize service delivery with established links between resources, operations, and citizens. Modern technology enables those connections with the help of sensors, mobile applications, and IoT-connected devices, thus allowing urban planners to capture data, manage resources, deploy services, and improve operations.
Smart cities rely on data-powered technology
Cities started growing rapidly during the 20th century with the power of technological innovation. Recent developments in big data technologies have led to cohesive urban systems with smart components that include:
- Communication networks, such as fiber-optic and wireless networks that connect various smart technologies of the city network.
- Governance and operational infrastructure that allows city management and administrative employees to coordinate activities, share data, and improve decision-making across different departments, such as waste management, transportation, public works, health, and emergency services.
- Internet of Things (IoT) devices embedded in streetlights, waste management systems, buildings, utilities, and transportation systems to enable real-time monitoring and control.
- Urban mobility systems that monitor traffic, parking, public transit, bike-sharing programs, and electric vehicle (EV) infrastructure.
- Energy management systems to distribute, monitor, and optimize services such as smart meters and smart grids.
- Waste management infrastructure, including sensors that monitor waste levels and optimize collection routes.
- Smart building systems that automate temperature, lighting, and ventilation to optimize energy consumption and improve air quality.
- Safety and security systems that leverage sensors, surveillance devices, and data analytics to monitor public spaces, detect criminal activity, and enable fast emergency responses.
The above components enable effective resource management and service delivery in the city. The success of these programs can be measured by evaluating service usage, monitoring key performance indicators, and leveraging sentiment analysis to gain insights directly from citizens.
Sentiment analysis can measure the success of smart city initiatives
Sentiment analysis is a machine-learning-powered branch of psychology that allows analysts to extract citizen emotions (or sentiments) from the public data generated in applications, forums, surveys, and social media.
Organizations can then use insights extracted from sentiment analysis to improve services in the following areas:
- Transportation
- Healthcare
- Crisis management
- Social services
- Education
- Law enforcement and public safety
- Environmental services and sustainability
- Tourism and culture
Let’s take the example of a new bus route. In the past, it took weeks (or even months) to gauge the effectiveness of an additional transit service. Today, applications that track riders and generate data that is suitable for sentiment analysis can instantly provide insights into the route’s success.
How sentiment analysis works
Sentiment analysis uses ML algorithms and lexicon-based approaches to evaluate opinions, sentiments, and subjectivity from texts and then label them as negative, positive, or neutral.
Data is first collected from dedicated transit applications and online via web scraping from social media, article and blog comments, reviews, and forums. Analysts can then process the data in two primary ways:
1. Supervised ML algorithms
The supervised ML algorithm method first uses a process to label texts as positive, negative, or neutral based on the sentiment expressed. Algorithms are then trained on the dataset to learn patterns and features that indicate the text’s sentiment.
The trained algorithm can then analyze new texts and make predictions based on previously learned patterns and characteristics.
2. Sentiment dictionary or “lexicon”
This method uses a sentiment dictionary or “lexicon” containing a word list with associated sentiment scores for individual words. The overall sentiment value of the text is then calculated by aggregating individual scores of all words in the text.
Applying sentiment analysis for smart city programs
Sentiment analysis can typically be used on any text, making it a valuable tool for gaining insights and optimizing the performance of a wide range of public services. Some examples currently in use include:
Transportation
Citizen feedback obtained from social media, surveys, and online reviews gives policy planners insights into which aspects of public transportation work well and which need improvement. Sentiment analysis can also be used to identify common issues such as overcrowding, delays, and cleanliness.
A recent paper outlined a methodology for collecting sentiment data on London’s public transportation system using a text-mining application. Researchers used Twitter’s Search API to gather tweets with specific hashtags, including #tfl (Transport for London) and #londonunderground, to extract opinions and sentiments expressed by citizens regarding service delivery, fares, and customer service.
Crisis management
Crises such as floods, fires, power outages, and other disruptive events can cause panic that spreads quickly throughout densely populated urban areas. Sentiment analysis can improve the management of those events in several ways, including:
- Early detection
- Monitoring
- Crisis mapping and situational awareness
- Resource allocation
- Public communication and messaging
Further, sentiment analysis can help governments better understand the situation in areas where language barriers exist. One example is the DARPA Low Resource Languages for Emergent Incidents (LORELEI) initiative that developed language processing technologies specifically for foreign or “low-resource” languages. Through analytical systems developed by the program, researchers gained valuable situational information using data extracted from news articles, social media posts, blogs, or audio recordings.
Public safety and law enforcement
Monitoring public spaces is considered one of the most effective ways to prevent crime. In this context, sentiment analysis improves public safety and law enforcement by:
- Identifying emerging conflicts/issues
- Improving response times
- Optimizing police services and resource allocation
- Providing targeted communication to the public
A real-world example of sentiment analysis to improve public safety was outlined in a paper that analyzed initiatives carried out by the Punjab Safe Cities Authority (PSCA) in Lahore, Pakistan. In the paper, researchers described how the PSCA used integrated command and control centers, 8,000+ cameras, and monitoring sensors to evaluate PSCA performance. The system also leveraged sentiment analysis mining publicly available data and concluded that citizens had a “high level of satisfaction” with the policing authority.
The importance of top-tier web scraping infrastructure
Although data needed for sentiment analysis can be gathered from internal apps and websites, collecting public information from the web makes analysis more effective and in-depth. To gather such data at scale, public institutions need modern web scraping technologies.
Web scraping is a data gathering method that allows collecting publicly available information from various sites. The process is technically complex and requires proxies to distribute requests, access geo-locked data, and bypass server blocks.
Though web intelligence is vital for businesses and a growing number of companies already use web scraping technologies to gather competitive insights, the public sector has been lagging behind up till now. To share knowledge and best practices, Oxylabs launched Project 4β, which provides academics, governments, educational institutions, and NGOs with pro bono educational materials, public web data gathering infrastructure, and other necessary resources.
That there’s plenty of space for public institutions to employ web scraping technologies for the common good and social progress. Big data should become an asset beyond just business needs.
