The Battle of the Suburbs in Canberra

The first step for data collection was to get a list of towncenters and suburbs which belong to them. It turned out that this dataset with an easy to use format like .csv or spreadsheet is not easily found online. The Wikipedia page of Suburbs of Canberra includes such information in the expandable tables near the bottom.

A small challenge here was that those foldable tabels are nested and therefore same information may occure at different hierarchy levels. After an cross examination between the webpage and the html inspector, I was able to narrow down the innerest talbe that containing the list of suburbs together with their corresponding towncenters.

To get the latitude and longitude information for each suburb, the geopy library demonstrated in previous labs is used.

A separate DataFrame df_town was created to include the latitude and longitude information for towncenters. After merging the geo location into main DataFrame df_can, it looks like below:

Based on the geo information of all the suburbs and towncenters, I created the following Folium map to visualize the towncenters and suburbs:

My first attempt to calculate the driving distance between suburb to towncenter was to use Google Distance Matrix API following this blog post. Unfortunately at the time of this study, there is no longer free options to use the Google Distance Matrix API. As an alternative, I explored the Haversine Distance as an alternative to the driving distance.

By iterating each suburb and towncenter and calling the haversine function, a distance matrix representing the Haversine Distance between suburbs and towncenters were created and stored into a DataFrame. An extra column Nearest hold the distance of a suburb to its closest towncenter was appended, although this nearest towncenter may not necessarily be the demographic one that a suburb belongs to. This nearest distance to towncenter information was merged into the master DataFraem df_can.

Again, the dataset for the real estate price for each suburb is not readily (or at least freely) available online. Domain provides a suburb profile page where you can enter the name of a suburb and serch its profile which happens to include a Market trends table. For example, following figure shows the property sales information under a section named Market trends found in suburb Banks:

The price of properties under interest are 3 or 4 bedrooms house for the young family who want to settle down in Canberra. A little bit of exploration on the URLs for each suburb revealed that the suffix part has a pattern of suburb-name/act/postcoe with special character in the suburb name like single quote or space being replaced by a - character. Using the webscraping techniques (requests and BeautifulSoup), I was able to extract the relevant field data from this table.

It is worth to note that not all suburbs have this Market trends table as it is based on the sales data in the past 12 months. Additionally even the table exists, it is not always to have the price for 3/4 bedroom house for the very same reason. I took care of these exceptions during the webscraping process and below I will explain how to handle the missing information for some suburbs.

Similar to previous labs covered in the course, I used the Foursquare API to explore the venues around each suburb and categorize them. As mentioned in the beginning, Canberra has a quite extended geography and a rather low population density, the radius to count the venues for each suburb was set to 2km. The returned venues were grouped by each suburb and the total number of venues for each suburb was added into the main DataFrame. This colcludes the end of main data collection and the master DataFrame df_can now looks like below:

From the above 3 plots, it is evident that there are more venues in North and South Canberras and the house price there are also much higher than suburbs in other towncenters. The distance for suburbs to closed towncenter however does not present a significant difference which indicate the well planned geo structure of Canberra suburbs.

K-means clustering is one of the most commonly used clustering techniques to perform unsupervised learning on datasets. It aims to partition the dataset into k clusters in which each sample belongs to the cluster with the nearest mean. The k-means clustering minimises the data sample variance within clusters and therefor is useful for this study to group suburbs with similar profiles like median house price or distance to towncenters.

To perform k-means clustering on the master dataset df_can, several irrelevant columns such as Towncenter, Postcode, Latitude, Longitude are dropped from the DataFrame. Then the relevant data are normalized first before the clustering. For k-means on the second venue category dataset, the number of different venue categories in each suburb are averaged first as a one-hot pre-processing.

To find the optimal number of clusters, the elbow method will be used where the optimal k value is decided when the quality of the clustering see a sharp drop. The tow figures below shows the distortion of the clusters on the two datasets and the optimal number of clusters are found to be 3 and 4 for the main and venue category datasets respectively.

The two figures below showed Folium maps and the clustered suburbs based on the main dataset the venue category dataset, together with towncenters. From the left figure, it is clear that most suburbs in North and South Canberra have been segmented into one category. The other cluster include those suburbs which are very close to towncenters but do not belong to the inner north or south Canberra. The rest suburbs which are located near the edge of each local towncenter make up the last category.

The right figure showing the suburbs clusters based on the venue category information tells a different story though. While suburbs in North and South Canberra still fall in one category, they are joined by some suburbs from the Tuggeranong area. The other obvious trend is that suburbs belonging to north and south towncenters seem to differ from the type of venues they host. This might be related to the development and poupulation growth shift in Canberra happened in recent years but will need some further analysis.

After clustering and segmenting all the Canberra suburbs based on the main data set, i.e. median price for 3 or 4 bedrooms houses, distance to nearest towncenters and number of venues within 2km radius, the same statitics were performed again on the suburbs grouped by clusters instead of towncenters. Cluster 2 includes most of the inner north and south suburbs in Canberra. These suburbs cover the parliamentary triangle of Canberra and many of the national galleries and institutions can be found here too. Therefore the number of venues are much higher than the other two clusters. The downside of these suburbs are the much higher median house prices as shown in the first figure. Cluster 1 suburbs mainly represent those close to local towncenters and in my opinion have the best trade-offs: surly the do not have as many venues as the inner ones but their house prices are also much lower. The last cluster 0 is made up by the suburbs near the edge of each local towncenters evidenced by the nearest distance to towncenters shown in the right most figure. They do not show many advantages compared to suburbs in the other two groups: for example, their distance to nearest towncenter almost doubled than those in cluster 1; the number of venues are much less while the median price for 3 or 4 bedrooms house showed a very similar profile to cluster 1 suburbs.