Data

I developed this app based on places that I like to visit when traveling: cafes with specialty coffee, breweries, and museums.

I use the European Coffee Trip website to make sure I get great coffee when I visit a new city, so their city guides served as the starting point for building the coffee dataset. I took a top-down approach to create the dataset. First, I extracted a list of cities by country and then extracted a list of cafes for each city. Once I had all the cafes, I was able to source cafe-level information (i.e., address, name, coordinates, etc.) from their respective page on the website. Additional queries were sent to Google Places API to fill in missing data where possible. The coffee dataset was sourced using the rvest, magrittr, stringr packages. The googleway package was used to send requests to the Google Places API.

In order to source data for breweries and museums, I created a dataset of unique cities. However, I needed to find the center point of each city (i.e., centroid). Initially, I calculated the centroid using the coordinates for all cafes. However, this wasn't an accurate measurement as not all cafes are located close to the city center. As a result, the center point was not always in the center of the city. For example, cities near the coast often had a center point located in the sea. To address this issue, I generated a bounding box for each city using the getbb function from the osmdata package. Centroids of each city were calculated using the mean of min/max latitude and longitude values. The following function was used to calculate the centroid coordinates:

bb_centroid <- function(x) {
    d <- as.data.frame(x)
    return(
        data.frame(
            lng = (d$min[1] + d$max[1]) / 2,
            lat = (d$min[2] + d$max[2]) / 2,
            stringsAsFactors = FALSE
        )
    )
}

Reference: this function can be found in the following script.

Using the list of unique cities, I used the coordinates to write Overpass API queries to source breweries and museums in those cities. OSM tags were selected using tagfinder.herokuapp.com and queries were wrapped into a function that could be used in a loop. Overpass Turbo was used to test and debug queries. Queries for breweries and museums were rewritten into functions that could be used in a batch querying script. All queries were structured to return nodes, ways, and relations. A 35000-meter search radius was used to limit the results around each city. These functions are listed below.

# overpass api query for finding breweries
brew$overpass$new_query <- function(radius = 35000, lat, lng, timeout = 750) {
    paste0(
        "[out:json][timeout:", timeout, "];",
        "(",
        "nwr",
        "['craft'='brewery']",
        "(around:", radius, ",", lat, ",", lng, ");",
        "nwr",
        "['amenity'='restaurant']",
        "['microbrewery'='yes']",
        "(around:", radius, ",", lat, ",", lng, ");",
        "nwr",
        "['amenity'='pub']",
        "['microbrewery'='yes']",
        "(around:", radius, ",", lat, ",", lng, ");",
        "nwr",
        "['amenity'='bar']",
        "['microbrewery'='yes']",
        "(around:", radius, ",", lat, ",", lng, ");",
        "nwr",
        "['building'='brewery']",
        "(around:", radius, ",", lat, ",", lng, ");",
        ");",
        "out;",
        ">;",
        "out skel qt;",
        sep = ""
    )
}

# overpass api query for finding museums
museums$new_query <- function(radius = 35000, lat, lng, timeout = 750) {
    paste0(
       "[out:json][timeout:", timeout, "];",
        "(",
        "nwr['tourism'='museum']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='history']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='art']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='railway']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='local']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='open_air']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='technology']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='nature']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='military']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='archaeological']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='local']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='transport']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='living_history']", "(around:", radius, ",", lat, ",", lng, ");",
        "nwr['museum'='children']", "(around:", radius, ",", lat, ",", lng, ");",
        ");",
        "out;",
        ">;",
        "out skel qt;",
        sep = ""
    )
}

The scripts for running queries to find breweries and museums in each city were run in RStudio cloud. API requests were sent using the httr package. Each script took several hours to run. All results were saved to json format. Data was extracted from json files using the jsonlite package. Not all queries returned the same data structure. Using the results for London, variables of interest were defined and used to evaluate missing variables in all data files. NA values were used for missing variables. The breweries and museums datasets were saved into separate files for additional geocoding.

Nominatim API queries were run to fill in missing information and geocoordinates. Using the OSM ID and type (e.g., way, node, relation), requests were sent to the lookup API using the httr package. The function used to build queries is listed below.

tools$new_request <- function(id, type = "N", format = "json") {
    prefix <- "https://nominatim.openstreetmap.org/lookup?"
    output <- paste0("&format=", format)
    return(paste0(prefix, "osm_ids=", type, id, output))
}

Each script took several hours to run. All column names were standardized to make it easier to merge the data later on. All scripts and data were pushed to the travel-app-data repo, and then pulled into the travel-app repo.

To prepare the data for use in the app, all duplicated rows were removed and locations without names were excluded. Locations with missing coordinates were also removed. All locations were plotted on a map. For locations that mistakenly appeared in countries outside Europe, those incorrect coordinates were fixed. The datasets were reduced to columns of interest (id, country, city, coords, name, place type) and then merged into a master dataset: all_european_places.RDS.

The master dataset was used to generate a series of summary objects, including the number of places by city. The places_by_city object is the primary data source for the app as it contains the number of places per type by city. This data is used to create travel recommendations, generate input elements and filters, and render the summary tables.

In total, eight datasets were created. Three of them are used in the final application and located in the data/ directory. The filenames and descriptions can be found in the following table. See the file data/data_1_prep.R to see how these files were generated.

travel_preferences <- function(weights, data) {

    #' Define a function that builds a blank user preferences object
    new_prefs <- function(data) {
        data.frame(
            id = data$id,
            city = data$city,
            country = data$country,
            lat = data$lat,
            lng = data$lng,
            data[, c("brewery", "cafe", "museum")] * 0,
            score = 0,
            stringsAsFactors = FALSE
        )
    }

    #' Create Required Objects (weights, references, and preferences)
    user_weights <- as.numeric(weights)
    refs <- as.matrix(data[, c("brewery", "cafe", "museum")])
    prefs <- new_prefs(data = data)

    #' Build a new score per city (weighted mean). I'm using my own
    #' weighted means formula in case I want to use the subscores in
    #' the app
    for (d in seq_len(NROW(refs))) {
        scores <- (refs[d, ] * user_weights)
        prefs[d, c("brewery", "cafe", "museum", "score")] <- cbind(
            rbind(scores),
            score = sum(scores) / sum(user_weights)
        )
    }

    #' Return
    return(prefs[order(prefs$score, decreasing = TRUE), ])
}