Neo4j

JavaScript

Python

GraphQL

DataViz

GRANDstack

Next.js

GRANDcast

Cypher

Spatial

Cloud

Travel

Data Visualization

Wherobots

Apache Sedona

Analytics

Java

Startups

React

Graph Visualization

Sigma.js

Neo4j Aura

GitHub

Data scraping

Low Code

Neo4

Chile

Conference

Building interactive graph visualizations with Sigma React - from basic setup to advanced customizations

A Look At Graph Visualization With Sigma React


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';

Every year with my family I head out into a nearby National Forest with a saw (except when I forget to bring it) and a National Forest Motor Vehicle User Map (MVUM) to find the annual Lyon family Christmas tree. I always wondered if I could analyze the data for the MVUMs to make my own map for hunting Christmas trees - so that's what I did for this map as part of the [30 Day Map Challenge.](https://github.com/johnymontana/30-day-map-challenge/tree/main)

I also did a bit of research using aerial imagery and QGIS to find areas that might have the perfect Christmas trees, then loaded the National Forest MVUM data in [Wherobots Cloud](https://wherobots.com/wherobots-cloud/) to find the forest service roads leading to the areas I annotated in QGIS with Spatial SQL.

<Image
  src={'/static/images/finding-tree/07-navigation-final.png'}
  alt={'Finding the perfect Christmas tree'}
  width={3300}
  height={2550}
/>

To follow along create a free account in [Wherobots Cloud](https://www.wherobots.services)

<Subscribe />

## Loading National Forest Motor Vehicle User Map Data

The US Forest Service publishes the National Forest Motor Vehicle User Map (MVUM) which shows roads and trails for accessing National Forest land. These maps are usually my primary navigation method when I'm driving into a National Forest. The USFS makes this map data available for download [here](https://data-usfs.hub.arcgis.com/datasets/motor-vehicle-use-map-roads-feature-layer/explore) as GeoJSON, CSV, or Shapefile.

I downloaded the Shapefile format, then uploaded it to Wherobots Cloud via the "Files" tab, which allows me to access this data via S3.

<Image
  src={'/static/images/finding-tree/files1.png'}
  alt={'Working with files in Wherobots Cloud'}
  width={3134}
  height={1700}
/>

Now we're ready to import that Shapefile into SedonaDB.

```py
# To import a Shapefile into SedonaDB we can use the built-in ShapefileReader which will
#   give us a SpatialRDD then convert that SpatialRDD to a Spatial DataFrame
# Then we create a new view called roads with this MVUM map data
# Finally we print the schema of the roads_df DataFrame

spatialRDD = ShapefileReader.readToGeometryRDD(sedona, S3_URL_ROADS_SHP)
roads_df = Adapter.toDf(spatialRDD, sedona)
roads_df.createOrReplaceTempView("roads")
roads_df.printSchema()
```

```md
root
|-- geometry: geometry (nullable = true)
|-- OBJECTID: string (nullable = true)
|-- RTE*CN: string (nullable = true)
|-- ID: string (nullable = true)
|-- NAME: string (nullable = true)
|-- BMP: string (nullable = true)
|-- EMP: string (nullable = true)
|-- SEG_LENGTH: string (nullable = true)
|-- GIS_MILES: string (nullable = true)
|-- SYMBOL: string (nullable = true)
|-- MVUM_SYMBO: string (nullable = true)
|-- JURISDICTI: string (nullable = true)
|-- OPERATIONA: string (nullable = true)
|-- SURFACETYP: string (nullable = true)
|-- SYSTEM: string (nullable = true)
|-- SEASONAL: string (nullable = true)
|-- PASSENGERV: string (nullable = true)
|-- PASSENGE_1: string (nullable = true)
|-- HIGHCLEARA: string (nullable = true)
|-- HIGHCLEA_1: string (nullable = true)
|-- TRUCK: string (nullable = true)
|-- TRUCK_DATE: string (nullable = true)
|-- BUS: string (nullable = true)
|-- BUS_DATESO: string (nullable = true)
|-- MOTORHOME: string (nullable = true)
|-- MOTORHOME*: string (nullable = true)
|-- FOURWD*GT5: string (nullable = true)
|-- FOURWD_G_1: string (nullable = true)
|-- TWOWD_GT50: string (nullable = true)
|-- TWOWD_GT_1: string (nullable = true)
|-- TRACKED_OH: string (nullable = true)
|-- TRACKED\_\_1: string (nullable = true)
|-- OTHER_OHV*: string (nullable = true)
|-- OTHER_OH_1: string (nullable = true)
|-- ATV: string (nullable = true)
|-- ATV_DATESO: string (nullable = true)
|-- MOTORCYCLE: string (nullable = true)
|-- MOTORCYC_1: string (nullable = true)
|-- OTHERWHEEL: string (nullable = true)
|-- OTHERWHE_1: string (nullable = true)
|-- TRACKED**2: string (nullable = true)
|-- TRACKED**3: string (nullable = true)
|-- OTHER_OH_2: string (nullable = true)
|-- OTHER_OH_3: string (nullable = true)
|-- ADMINORG: string (nullable = true)
|-- SECURITYID: string (nullable = true)
|-- DISTRICTNA: string (nullable = true)
|-- FORESTNAME: string (nullable = true)
|-- FIELD_ID: string (nullable = true)
|-- SBS_SYMBOL: string (nullable = true)
|-- ROUTESTATU: string (nullable = true)
|-- GLOBALID: string (nullable = true)
|-- TA_SYMBOL: string (nullable = true)
|-- SHAPE: string (nullable = true)
|-- SHAPELEN: string (nullable = true)
```

Now that we've created a view with our roads data we can use Spatial SQL to query our MVUM data.

```py
sedona.sql("SELECT COUNT(*) AS num FROM roads").show()
```

```md
+------+
| num  |
+------+
|151457|
+------+
```

Since we're going to be hunting for Christmas trees near Santa, Idaho we know we're only interested in roads within the Idaho Panhandle National Forests.

```py
idaho_df = sedona.sql("""
SELECT geometry, NAME as name, FORESTNAME AS forest, SEASONAL AS seasonal, SYMBOL AS symbol FROM roads
WHERE FORESTNAME = 'Idaho Panhandle National Forests'
""")
idaho_df.createOrReplaceTempView("idaho")
idaho_df.show(5)
```

```md
+--------------------+-----------------+--------------------+--------+------+
| geometry           | name            | forest             |seasonal|symbol|
+--------------------+-----------------+--------------------+--------+------+
|LINESTRING (-115....| DOMINION CREEK  |Idaho Panhandle N...|yearlong|     3|
|LINESTRING (-116....| PLACER RIDGE A  |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| RAINEY HILL     |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-117....| URANIUM         |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....|HAMILTON MOUNTAIN|Idaho Panhandle N...|yearlong|     1|
+--------------------+-----------------+--------------------+--------+------+
```

```py
SedonaKepler.create_map(idaho_df, name="Roads")
```

<Image
  src={'/static/images/finding-tree/kepler_roads.png'}
  alt={'Visualizing roads using SedonaKepler'}
  width={1530}
  height={806}
/>

That's a lot of roads! How are we going to figure out where the best Christmas trees are? Let's look at aerial imagery of the area to find what look like nice tree stands to help narrow down the hunt a bit. We'll use my favorite desktop GIS tool QGIS for this.

## Annotating Aerial Imagery In QGIS

QGIS by default includes tile layers from OpenStreetMap, but we can also add other tile server connections to bring in tile layers from other providers. [This StackExchange answer](https://gis.stackexchange.com/questions/20191/adding-basemaps-in-qgis/217670#217670) describes how to add aerial imagery from Google and Bing to QGIS.

<div style={{ margin: 'auto', maxWidth: '500px' }}>
  <Image
    src={'/static/images/finding-tree/qgis_newtiles.png'}
    alt={'Adding a new tile server to QGIS'}
    width={974}
    height={1178}
  />
</div>

<div style={{ margin: 'auto', maxWidth: '500px' }}>
  <Image
    src={'/static/images/finding-tree/qgis_google.png'}
    alt={'Adding Google aerial imagery'}
    width={1496}
    height={1300}
  />
</div>

Once we add the aerial imagery tile layer we can start to explore the area where
we hope to find some great Christmas trees, in this case in the Idaho Panhandle National
Forests. We're looking for nice heavy green tree stands, not too much vertical variation
for easy access and not near streams and rivers or hiking trails.

Let's add a new layer to our QGIS project so we can keep track of where these tree stands are. I chose a Geopackage layer with Polygon geometry because we're going to draw polygons over the aerial imagery to indicate places that we think will be good for potential trees.

<div style={{ margin: 'auto', maxWidth: '500px' }}>
  <Image
    src={'/static/images/finding-tree/qgis_newlayer.png'}
    alt={'Adding a new layer in QGIS'}
    width={1320}
    height={1652}
  />
</div>

Right-click on the layer and choose "Toggle editing" then click the icon to create
a new polygon. We can now draw polygons over the aerial imagery, each new polygon
will be added as a new feature to our new Geopackage layer. I didn't create any other
fields so the `fid` is a sequence that will be autogenerated. If you wanted to keep
notes about why you thought this area looked good you could do that by adding another
field to the layer.

<Image
  src={'/static/images/finding-tree/qgis_polygon.png'}
  alt={'Adding polygon features'}
  width={1830}
  height={1262}
/>

Once we've annotated a few areas (I found 5 or 6) we can right click on our layer and select "Save as". I exported this layer as GeoJSON, to a file called `idaho_treestands.geojson`.

<div style={{ margin: 'auto', maxWidth: '500px' }}>
  <Image
    src={'/static/images/finding-tree/qgis_exportgeojson.png'}
    alt={'LinkedIn Post'}
    width={1406}
    height={1544}
  />
</div>

## Finding Roads To Our Tree Stands With SedonaDB

Now that we've exported our polygon annotations as GeoJSON it's time to import that data into SedonaDB along with our road data so we can the Forest Service roads that will take us to our tree stands. Return to the "Files" tab in Wherobots Cloud and upload the GeoJSON file we just exported from QGIS.

<Image
  src={'/static/images/finding-tree/files2.png'}
  alt={'The Wherobots Files interface'}
  width={2674}
  height={1096}
/>

To import GeoJSON into SedonaDB we can use Spark's built-in JSON import functionality if we first define the schema used by GeoJSON.

```py
schema = "type string, name string, crs string, features array<struct<type string, geometry string, properties map<string, string>>>"
```

```py
tree_df = sedona.read.option('multiline', True).json(S3_URL_TREES, schema=schema).selectExpr("explode(features) as features").select("features.*").withColumn("geometry", expr("ST_GeomFromGeoJson(geometry)")).withColumn("fid", expr("properties['fid']")).drop("properties").drop("type")
tree_df.createOrReplaceTempView("trees")
tree_df.show(5)
```

```md
+--------------------+---+
| geometry           |fid|
+--------------------+---+
|POLYGON ((-116.52...|  1|
|POLYGON ((-116.54...|  2|
|POLYGON ((-116.12...|  3|
|POLYGON ((-116.16...|  4|
|POLYGON ((-116.09...|  5|
+--------------------+---+
only showing top 5 rows
```

```py
SedonaKepler.create_map(tree_df, name="Tree Stands")
```

<Image
  src={'/static/images/finding-tree/kepler_trees.png'}
  alt={'Visualizing the tree layer with SedonaKepler'}
  width={2008}
  height={822}
/>

As we can see above I found 6 possible tree stands I want to explore. Now we can do a spatial join operation to find the Forest Service roads nearest these tree stands. We could either use the [`ST_Distance` Spatial SQL function](https://docs.wherobots.services/1.2.0/references/sedonadb/vector-data/Function/?h=st_distance#st_distance) to find the closest roads or perhaps use [the `ST_Intersects` predicate](https://docs.wherobots.services/1.2.0/references/sedonadb/vector-data/Predicate/#st_intersects) if we just want to find the roads that go through the tree stands we annotated.

```py
routes1_df = sedona.sql("""
SELECT idaho.geometry, name, forest, seasonal, symbol
FROM idaho, trees
WHERE ST_Intersects(trees.geometry, idaho.geometry)
""")

routes1_df.show()
```

```md
+--------------------+----------------+--------------------+--------+------+
| geometry           | name           | forest             |seasonal|symbol|
+--------------------+----------------+--------------------+--------+------+
|LINESTRING (-116....| PALOUSE DIVIDE |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| CLARKIA MARBLE |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....|BONNEVILLE POWER|Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| SWAN PEAK      |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| CORNWALL CREEK |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| LACEY CREEK    |Idaho Panhandle N...|yearlong|     1|
|LINESTRING (-116....| CRANBERRY      |Idaho Panhandle N...|yearlong|     1|
+--------------------+----------------+--------------------+--------+------+
```

```py
map = SedonaKepler.create_map(routes1_df, name="routes")
SedonaKepler.add_df(map, tree_df, name="Tree Stands")
```

<Image
  src={'/static/images/finding-tree/kepler_route1.png'}
  alt={'LinkedIn Post'}
  width={2000}
  height={816}
/>

As we can see above most of our tree stands have Forest Service roads leading right though them. We want to expand our route a bit though - we want to know what roads we need to take to turn onto the roads that intersect our tree stands. To do this we can use a SQL subquery to find roads that either intersect or touch the roads that intersect our tree stands.

```py
routes_df = sedona.sql("""
SELECT DISTINCT idaho.geometry, idaho.name, idaho.forest, idaho.seasonal
FROM (
    SELECT idaho.geometry, name, forest, seasonal, symbol
    FROM idaho, trees
    WHERE ST_Intersects(trees.geometry, idaho.geometry)) AS routes, idaho
WHERE ST_Touches(idaho.geometry, routes.geometry)
""")
routes_df.show()
```

```md
+--------------------+------------------+--------------------+--------+
| geometry           | name             | forest             |seasonal|
+--------------------+------------------+--------------------+--------+
|LINESTRING (-116....| CLARKIA MARBLE   |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CHARLIE CREEK    |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CRANBERRY        |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CAMP 3           |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| PALOUSE DIVIDE   |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| 3332 E           |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| HOMESTEAD CREEK  |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| PALOUSE DIVIDE   |Idaho Panhandle N...|seasonal|
|LINESTRING (-116....| EAGLE CREEK      |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| BONNEVILLE POWER |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....|CHARLIE CONNECTION|Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CRANE            |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| 3332 F           |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| HOODOO           |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| LACEY CREEK      |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CLARKIA MARBLE   |Idaho Panhandle N...|yearlong|
|LINESTRING (-116....| CORNWALL CREEK   |Idaho Panhandle N...|yearlong|
+--------------------+------------------+--------------------+--------+
```

<Image
    src={'/static/images/finding-tree/kepler_routes.png'}
    alt={'Visualizing routes with SedonaKepler'}
    width={2002}
    height={812}
  />

That gives us a better indication of how to navigate to our tree stands! Next, we'll export our route data to GeoJSON to load back into QGIS to combine with other data from our MVUM dataset to create a navigation map so we can find our Christmas trees!

```py
routes_gdf = geopandas.GeoDataFrame(routes_df.toPandas(), geometry="geometry")
routes_gdf.to_file('routes.geojson', driver='GeoJSON')
````

And here's the final map:

<Image
  src={'/static/images/finding-tree/07-navigation-final.png'}
  alt={'Christmas Tree Hunting Near Santa, Idaho'}
  width={3300}
  height={2550}
/>

## Resources

- [Code on GitHub](https://github.com/johnymontana/30-day-map-challenge/blob/main/notebooks/07-navigation.ipynb)
- [Wherobots Cloud](https://wherobots.com/wherobots-cloud/)


Using aerial imagery and Spatial SQL to explore Forest Service road networks

Finding The Perfect Christmas Tree With USFS Motor Vehicle User Map Data, QGIS, & SedonaDB

My take on the exciting world of spatial database technology plus a look at getting started with Apache Sedona.

Why I Joined Wherobots Plus A Look At Apache Sedona


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';

## Plus Writing Spatial SQL Queries With AWS Athena

In this post we take a look at the recently released public map dataset from the Overture Maps Foundation. We'll see how to select a subset of the data using AWS Athena, how to import the map data into a Neo4j graph database using Neo4j Aura's no-code visual import and data modeling tooling, and finally we'll see how to leverage other no & low-code tooling to create a dashboard using the Neodash graph app and GraphQL API using the Neo4j GraphQL Toolbox.

<Subscribe />

## Overture Maps Foundation Releases Public Dataset

The [Overture Maps Foundation](https://overturemaps.org/) is a collaboration among Meta, Microsoft, TomTom, Amazon, and others with the goal of building a public dataset of map data. It's a Linux Foundation project, similar in structure to the GraphQL Foundation which is a group I'm more familiar with having previously served as Neo4j's representative to the GraphQL Foundation for a few years. 

Overture's goal in their own words is:

> Powering current and next-generation map products by creating reliable, easy-to-use, and interoperable open map data.

<div style={{ margin: 'auto', maxWidth: '500px' }}>
  <Image
    src={'/static/images/overture-graph/overture1.png'}
    alt={'The Overture Maps Foundation'}
    width={518}
    height={428}
  />
</div>

Overture released the first alpha version of their public dataset on July 26th, 2023 so I thought this would be a good time to take a look at the structure of the data in the public dataset as well as see how we can import and analyze the data using Neo4j.

The Overture data consists of four "themes": _places_, _transport_, _admins_, and _buildings_ and is published as Parquet files available via AWS or Azure. Previously I've used Python's pandas library and the Neo4j Python Driver to parse Parquet files into Python dataframes and then import into Neo4j using Cypher (you can see an example of that approach [here where I import the Daylight Earth Table OpenStreetMap distribution into Neo4j](https://github.com/johnymontana/daylight-earth-graph/blob/main/POI_import.ipynb) from Parquet files) but for this project I wanted to stay as code-free as possible and instead use AWS Athena's serverless SQL engine to parse and query the Overture Parquet files then use Neo4j's visual data import tool. This did require that I write a bit of SQL.

You can find more detailed information about each theme in the Overture Maps documentation [here.](https://docs.overturemaps.org/)

## Querying Overture Maps Data With AWS Athena

AWS Athena is a serverless SQL engine that will allow us to execute SQL queries on the Parquet files distributed by Overture.

First, in AWS Athena we need to create a table. I'll start with creating the `places` table, which maps to the Overture places Parquet theme. To create the table in Athena I'll need to specify the schema used for the places theme.

```sql
CREATE EXTERNAL TABLE `places`(
  `id` string,
  `updateTime` string,
  `version` int,
  `names` map<string,array<map<string,string>>>,
  `categories` struct<main:string,alternate:array<string>>,
  `confidence` double,
  `websites` array<string>,
  `socials` array<string>,
  `emails` array<string>,
  `phones` array<string>,
  `brand` struct<names:map<string,array<map<string,string>>>,wikidata:string>,
  `addresses` array<map<string,string>>,
  `sources` array<map<string,string>>,
  `bbox` struct<minX:double,maxX:double,minY:double,maxY:double>,
  `geometry` binary)
PARTITIONED BY (
  `type` varchar(5))
STORED AS PARQUET
LOCATION
  's3://overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=places'
```

And then to trigger building the table.

```sql
MSCK REPAIR TABLE `places`
```

Now that we've created the `places` table, let's write a SQL query to fetch a subset of the Overture places data based on some bouding box. We'll download the results as a CSV file and then use the no-code Neo4j Import tool to create our graph in Neo4j Aura by mapping fields of the CSV file to the property graph model we wish to create using a visual diagramming interface. It's really quite slick!

Let's subset the data based on the bounds of Clark County, Nevada in the US (home of Las Vegas). We can do that using a `bbox` predicate that specifies the lower-left and upper-right coordinates of our bounding box. Some of the attributes in the `places` table are json data so we'll also parse those fields as json and extract the relevant scalar values such as the name, category, and address components using `json_extract` and `json_array_get` SQL functions.

For the places theme each row will contain a point geometry, stored efficiently in the binary WKB (Well Known Binary) format. We'll use the `ST_GeomFromBinary` spatial SQL function to convert that geometry field to WKT (Well Known Text) format and then the `ST_X` and `ST_Y` functions to retrieve the point's longitude and latitude.

```sql
SELECT
    id, updatetime, version, confidence, websites, socials, emails, phones, type,
    ST_X(ST_GeomFromBinary(geometry)) as longitude,
    ST_Y(ST_GeomFromBinary(geometry)) as latitude,
    json_extract_scalar(cast(categories as json), '$.main') as category,
    json_extract(cast(categories as json), '$.alternate') as alt_categories,
    json_extract_scalar(cast(names as json), '$.common[0].value') as name,
    json_extract(json_array_get(cast(sources as json), 0), '$.dataset') as source_dataset,
    json_extract(json_array_get(cast(sources as json), 0), '$.recordid') as source_recordid,
    json_extract(json_array_get(cast(addresses as json), 0), '$.freeform') as street_address,
    json_extract(json_array_get(cast(addresses as json), 0), '$.locality') as locality,
    json_extract(json_array_get(cast(addresses as json), 0), '$.postcode') as postcode,
    json_extract(json_array_get(cast(addresses as json), 0), '$.region') as region,
    json_extract(json_array_get(cast(addresses as json), 0), '$.country') as country
FROM
    places
WHERE
        bbox.minX >  -115.896925
    AND bbox.maxX <  -114.042819
    AND bbox.minY >  35.001857
    AND bbox.maxY <  36.853662
```

<Image
  src={'/static/images/overture-graph/athena3.png'}
  alt={'Querying the Overture Maps places theme with Amazon Athena'}
  width={2954}
  height={1856}
/>

In the AWS Athena console we can select "Download data" to download a CSV file of the results. The file is 26Mb and has about 75k rows.

| "id"                                   | "updatetime"              | "version" | "confidence"          | "websites"                                 | "socials"                                    | "emails" | "phones"         | "type"  | "longitude"    | "latitude"   | "category"       | "alt_categories"                         | "name"                          | "source_dataset" | "source_recordid"      | "street_address"                | "locality"         | "postcode"       | "region" | "country" |
| -------------------------------------- | ------------------------- | --------- | --------------------- | ------------------------------------------ | -------------------------------------------- | -------- | ---------------- | ------- | -------------- | ------------ | ---------------- | ---------------------------------------- | ------------------------------- | ---------------- | ---------------------- | ------------------------------- | ------------------ | ---------------- | -------- | --------- |
| "tmp_9A5E9AA9E18DB0FD734E236DBEEB73B5" | "2023-07-24T00:00:00.000" | "0"       | "0.5052741765975952"  |                                            | "[https://www.facebook.com/140308446009738]" |          |                  | "place" | "-114.3012956" | "35.5715668" | "restaurant"     | "[""landmark_and_historical_building""]" | "Lake Mohave Ranchos Airport"   | """meta"""       | """140308446009738"""  |                                 |                    |                  |          |
| "tmp_7BCFD4C7C12B83CD5355AE15F5A3F262" | "2023-07-24T00:00:00.000" | "0"       | "0.6"                 |                                            |                                              |          |                  | "place" | "-115.0231676" | "36.0496907" |                  | "null"                                   | "Local Vape"                    | """msft"""       | """1970324838862842""" | """7685 Commercial Way Ste I""" | """Henderson"""    | """89011"""      | """NV""" | """US"""  |
| "tmp_FF0104AF0B2048DCD02320B4A096A398" | "2023-07-24T00:00:00.000" | "0"       | "0.26928654313087463" | "[http://ombremonai.com]"                  | "[https://www.facebook.com/108034485157293]" |          | "[+17029101015]" | "place" | "-115.28665"   | "36.15881"   | "beauty_and_spa" | "null"                                   | "Ombré Monai"                   | """meta"""       | """108034485157293"""  | """8751 W Charleston Blvd"""    | """Las Vegas"""    | """89117-5453""" | """NV""" | """US"""  |
| "tmp_D57D0159A70E965E96572CF6D596F1C1" | "2023-07-24T00:00:00.000" | "0"       | "0.8260171413421631"  | "[https://nevadasouthernlivesteamers.com]" | "[https://www.facebook.com/475380855987545]" |          | "[+17023757089]" | "place" | "-114.8539342" | "35.9706829" | "museum"         | "[""history_museum"",""train_station""]" | "Nevada Southern Live Steamers" | """meta"""       | """475380855987545"""  | """601 Yucca St"""              | """Boulder City""" | """89005"""      | """NV""" | """US"""  |

With our CSV file of points of interest downloaded, let's head over to [Neo4j Aura](https://dev.neo4j.com/neo4j-aura) and spin up a free Neo4j instance in the cloud to analyze our place data. After signing in and creating a free tier Neo4j Aura instance we'll use the "Import" tab to create a property graph data model mapped to the places data in our CSV file.

<div style={{ margin: 'auto', maxWidth: '750px' }}>
  <Image
    src={'/static/images/overture-graph/workspace1.png'}
    alt={"Using Neo4j Aura's Workspace developer tooling"}
    width={1386}
    height={606}
  />
</div>

## Data Import Tool

Neo4j Workspace's Import tool is designed around the idea of sketching your property graph data model in an interactive modern diagramming UI and mapping pieces of the data model diagram to fields in your CSV files.

I had worked on [a similar web-based visual graph data import tool prototype](https://github.com/neo4j-contrib/neo4j-csv-import-web) when I first joined Neo4j 8 years ago so it's great to see this functionality available directly in Neo4j Aura.

We can start either by beginning to sketch our property graph data model as we would on a whiteboard, or by selecting the CSV file we want to import. We'll start by selecting the CSV file we downloaded from AWS Athena in the previous step.

<Image
  src={'/static/images/overture-graph/workspace2.png'}
  alt={
    'Using the Neo4j Import Tool to load Overture Maps point of interest data into the Neo4j graph database.'
  }
  width={2524}
  height={1178}
/>

No data is sent to the server, but Import has parsed our CSV file and shows a preview of the values from the first row that we can use to reference when sketching our whiteboard graph model.

<Image
  src={'/static/images/overture-graph/workspace3.png'}
  alt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2516}
  height={1176}
/>

If we select the "Add node label" button we can add our first node to the diagram. Let's use this first node label to represent the places of interest themselves so we'll call this node label `PointOfInterest`.

<Image
  src={'/static/images/overture-graph/workspace4.png'}
  slt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2516}
  height={1178}
/>

Next, we'll want to specify the properties to include on our `PointOfInterest` nodes from the CSV file. Some of the fields from our CSV file will be extracted out into nodes and relationships of their own (like category, source, and location information) so we don't want to include _all_ fields from the CSV.

We also choose the type of each field and importantly also specify which field defines uniqueness for the node. In other words, which field is the "id" value. In the relational database world this is similar to the concept of "primary key". We need to specify this property to be able to define relationships in our graph.

<Image
  src={'/static/images/overture-graph/workspace5.png'}
  alt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2520}
  height={1168}
/>

Adding relationships to the data model is as simple as dragging out from the halo around a node.

<Image
  src={'/static/images/overture-graph/workspace6.png'}
  alt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2510}
  height={1170}
/>

In the labeled property graph model used by Neo4j every relationship has both a direction and single "type" that describes the relationship between the two node labels that it connects. We can add the relationship type by clicking on the dashed relationship line.

Since this relationship connects the `PointOfInterest` and `Category` node labels let's define the relationship type as `IN_CATEGORY`. By convention [PascalCase](https://en.wiktionary.org/wiki/Pascal_case) is used when defining node lables and ALL_CAPS_SNAKE_CASE is used for relationship types. You can find more Cypher style guidelines [here](https://neo4j.com/developer/cypher/style-guide/).

<Image
  src={'/static/images/overture-graph/workspace7.png'}
  alt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2514}
  height={1178}
/>

To enable the Neo4j Import tool to properly connect nodes we need to specify which fields hold the id values for the "from" and "to" nodes. In this case we have an `id` property for the `PointOfInterest` node which specifies the "from" node and the `category` property value specifies the "to" node.

Notice the green checkmarks that appear as we start to fill out more pieces of the property graph model indicating that we have specified the necessary field mappings.

<Image
  src={'/static/images/overture-graph/workspace8.png'}
  alt={
    'Sketching a property graph data model using Neo4j Workspace Import tool'
  }
  width={2512}
  height={1174}
/>

Next, we continue building up our property graph model, mapping the entities to node labels and the fields that describe them to properties in the graph data model.

<Image
  src={'/static/images/overture-graph/import2.png'}
  alt={
    'Using the Neo4j Import Tool to load Overture Maps point of interest data into the Neo4j graph database.'
  }
  width={3140}
  height={1950}
/>

Here's the final graph model that I ended up with for the Overture dataset.

<Image
  src={'/static/images/overture-graph/import4.png'}
  alt={
    'Using the Neo4j Import Tool to load Overture Maps point of interest data into the Neo4j graph database.'
  }
  width={3122}
  height={1670}
/>

Once we've fully defined our graph data model and mapped to the associated fields of our CSV file we're ready to run the import into our Neo4j Aura instance.

To start the import process we'll click the "Run import" button. Behind the scenes the Neo4j Import tool will generate Cypher queries to create property constraints and then iterate over the data in our CSV file in batches to load data into our Neo4j Aura instance.

<Image
  src={'/static/images/overture-graph/import5.png'}
  alt={
    'Using the Neo4j Import Tool to load Overture Maps point of interest data into the Neo4j graph database.'
  }
  width={1836}
  height={1816}
/>

## What Now?

Now that we've loaded our geospatial data into Neo4j what can we do with it?

We can use the power of Cypher to query for points of interest near us, within certain categories for example. Or more complex queries for location-aware recommendations. For example, let's find all the wineries near me:


```cypher
MATCH (p:PointOfInterest)-[:IN_CATEGORY]->(c:Category {name: "Winery"})
WHERE point.distance(p.location, {latitude: 37.594, longitude: -122.364}) < 100
RETURN p
```

But keeping with our theme of leveraging Neo4j's low-code tools what else do we have at our disposal? I've previously covered Neo4j Bloom which is a great tool for exploring graphs visually, so let's skip Bloom for now and take a look at two of Neo4j's other low-code tools: Neodash and the Neo4j GraphQL Toolbox.

__Building Graph Dashboards With Neodash__

One of the easiest ways to work with data in Neo4j without writing code is to use [NeoDash](https://neodash.graphapp.io/) - a tool from the Neo4j Labs team for building standalone dashboards backed by Neo4j. 

Neodash has support for creating maps and integrating spatial data into dashboards. Here we've just set up a very basic map view so we can explore the Overture data visually.

<Image
  src={'/static/images/overture-graph/neodash1.png'}
  alt={'Creating a dashboard with the Neodash graph app for Neo4j.'}
  width={1174}
  height={553}
/>

__Put The Graph In GraphQL With The Neo4j GraphQL Toolbox__

<Image
  src={'/static/images/overture-graph/graphql1.png'}
  alt={'Neo4j GraphQL Toolbox'}
  width={2421}
  height={1515}
/>

Another low-code tool from Neo4j is the [Neo4j GraphQL Toolbox](https://graphql-toolbox.neo4j.io/) which allows you to create a GraphQL API backed by Neo4j without writing any code. We can just point the Toolbox at our Neo4j Aura instance and generate a fully functional GraphQL API.


This post is just a starting point for working with the Overture maps dataset in Neo4j. It's exciting to see what Overture is doing and I'll definitely be making use of their dataset for future projects. 

## Resources

- [Accessing Overture Maps' Global Dataset using Athena (AWS)](https://feyeandal.me/blog/access_overture_data_using_athena)
- [Athena setup SQL for Overture (all themes)](https://github.com/OvertureMaps/data/blob/main/athena_setup_queries.sql#L77)
- [Extracting data from JSON (AWS Athena Docs)](https://docs.aws.amazon.com/athena/latest/ug/extracting-data-from-JSON.html)
- [Geospatial SQL functions available in Trino](https://trino.io/docs/current/functions/geospatial.html)
- [Bounding Boxes for all US Counties](https://trino.io/docs/current/functions/geospatial.html)
- [Spatial Cypher cheat sheet]()


A look at loading points of interest from the Overture Maps public dataset into the Neo4j graph database, plus writing Spatial SQL queries with AWS Athena.

Importing Overture Maps Data Into Neo4j

A resource for working with geospatial data in Neo4j.

Introducing The Spatial Cypher Cheat Sheet


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

In this post we explore some techniques for working with geospatial data in Neo4j. We will cover some basic spatial Cypher functions, spatial search, routing algorithms, and different methods of importing geospatial data into Neo4j. Be sure to sign up for my newsletter to be notified of new blog posts like these:

<Subscribe />

This post is based on my talk at [NODES 2022](https://dev.neo4j.com/nodes-videos), so you can watch the video recording instead:

<YouTube youTubeId="-fs8ozxKklQ" />

## Spatial Cypher Functions And The `Point` Type

Native geospatial functionality in Neo4j is based on support for a native `Point` type. The `Point` data type can be used to represent point data using either 3D or 2D geographic (WGS84) or cartesian coordinates reference systems (CRS). Points (or arrays of points) can be stored as properties on nodes or relationships in Neo4j. Cypher contains [several functions for working with points](https://neo4j.com/docs/cypher-manual/current/syntax/spatial/), notably:

- `point()` - for creating `Point` instances
- `point.distance()` - calculates distance between two points. Can also be used for radius distance search when used in a `WHERE` predicate
- `point.withinBBox()` - to find points within a given bounding box

Point instances can be created using the `point()` function in Cypher and passing an object that specifies latitude, longitude, and optionally height (when using geographic CRS) or x, y and optionally z (when using cartesian CRS). In this article we will focus on the 2D geographic CRS (using latitude an longitude).

Here we create a point instance by specifying latitude and longitude:

```Cypher
RETURN point(
  {latitude:49.38713, longitude:12.12711}
)
```

```md
╒════════════════════════════════════════════════╕
│"point({latitude:49.38713, longitude:12.12711})"│
╞════════════════════════════════════════════════╡
│point({srid:4326, x:12.12711, y:49.38713})      │
└────────────────────────────────────────────────┘
```

To set a point property value we can use the `SET` Cypher command. Here we create a new `PointOfInterest` node representing a bakery and set the `location` property to a point value:

```Cypher
CREATE (p:PointOfInterest)
SET p.name = "West Side Bakery",
p.location = point(
  {latitude:49.38713, longitude:12.12711}
)
RETURN p
```

<div style={{  margin: 'auto' }}>
  <Image
  src={'/static/images/nodes2022/node.png'}
  alt={'Creating a node with a location point property'}
  height={277}
  width={551}
/>


</div>


We can use the `point.distance()` function to search for our bakery. Here we search for nodes with the `PointOfInterest` label that are within 200 meters of a given latitude and longitude:

```Cypher
MATCH (p:PointOfInterest)
WHERE point.distance(
  p.location, point({latitude:49.38713, longitude:12.12711})
) < 200
RETURN p
```

Let's look at some real-world data to see how we can implement spatial search functionality in a simple web map application.

## Spatial Search With Neo4j

A common feature for web applications that deal with geospatial data involves spatial search: finding things near other things or within the bounds of a box or perhaps within a polygon. Let's see how we can use Neo4j to search for points of interest on a map using Leaflet.js for the following types of spatial search:

1. Radius distance search
2. Within Bounding Box
3. Within Polygon

<Image
  src={'/static/images/nodes2022/spatial_search.png'}
  alt={
    'Spatial search with Neo4j - radius distance, bounding box, and point in polygon'
  }
  width={1102}
  height={950}
/>

Code [on GitHub: johnymontana/geospatial-graph-demos.](https://github.com/johnymontana/geospatial-graph-demos)

I wanted to use a real world dataset, so I pulled in points of interest from the [Daylight Earth Table](https://daylightmap.org/earth/) distribution of OpenStreetMap. The Daylight Earth Table is based on the Daylight OpenStreetMap distribution, but adds a data schema that classifies OpenStreetMap tags into a three level ontology: theme, class, and subclass. We'll be using the "poi" theme.

<Image
  src={'/static/images/nodes2022/daylight.png'}
  alt={'The Daylight Earth Table OpenStreetMap distribution'}
  width={1395}
  height={843}
/>

Currently there are almost 39 million points of interest in the dataset. The Daylight Earth Table is available in Parquet files stored in S3 so I used the `pandas` Python library to read the Parquet files from S3 into a DataFrame. By converting the DataFrame to Python dicts and passing in batches as Cypher parameters we can import the dataset in a few minutes.

```Python
# Here we define a function to split our DataFrame into batches for import

def insert_data(query, rows, batch_size=10000):
    total = 0
    batch = 0
    start = time.time()
    result = None

    while batch * batch_size < len(rows):
        res = conn.query(query, parameters = {'rows': rows[batch*batch_size:(batch+1)*batch_size].to_dict('records')}, db="osmpois")
        total += res[0]['total']
        batch += 1
        result = {"total": total, "batches": batch, "time": time.time()-start}
        print(result)
    return result
```

```Cypher
def insert_pois(rows, batch_size=10000):
    print(rows)
    query = '''
    UNWIND $rows AS row
    CREATE (p:PointOfInterest {name: row.names.local})
    CREATE (g:Geometry)
    SET g.location = point({latitude: toFloat(row.point[1]), longitude: toFloat(row.point[0]) })
    CREATE (g)<-[:HAS_GEOMETRY]-(p)
    SET g:Point
    CREATE (t:Tags)
    SET t += row.original_tags_dict
    CREATE (p)-[:HAS_TAGS]->(t)
    WITH *
    CALL apoc.create.addLabels(p, [row.class, row.subclass]) YIELD node
    RETURN COUNT(*) AS total
    '''
    return insert_data(query, rows, batch_size)
```

<Image
  src={'/static/images/nodes2022/daylight_notebook.png'}
  alt={
    'Importing the Daylight Earth Table OpenStreetMap distribution into Neo4j'
  }
  width={1126}
  height={757}
/>

See the code on GitHub: [johnymontana/daylight-earth-graph](https://github.com/johnymontana/daylight-earth-graph)

### Radius Distance Search

Now that we have all points of interest loaded into Neo4j we'll create a point index to enable fast index-backed spatial search operations.

```Cypher
CREATE POINT INDEX poiIndex
FOR (p:Point) ON (p.location)
```

Our first spatial search type is radius distance search: finding all points of interest within a given distance of a point.

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/radius_search.png'}
  alt={'Radius distance search with Neo4j'}
  height={323}
  width={520}
/>
</div>

Here we search for all points of interest within 200 meters of a point in Northern California.

```Cypher
PROFILE MATCH (p:Point)
WHERE
point.distance(p.location, point({latitude: 37.563434, longitude:-122.322255})) < 200
RETURN p{.*} AS point
```

By prepending `PROFILE` to our Cypher query we will render the database query plan as well as basic performance metrics for the query, such as memory usage, execution time, number of rows at each operation, and db hits. We can see that our point index is used resulting in an execution time of 1-2ms. Compare to the execution plan on the left which does not use an index and rather must scan every point of interest and takes about 4 seconds to execute. When we're dealing with millions of nodes it behooves us to make sure our spatial search operations are using an index for best performance.

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/radius_index_plan.png'}
  alt={'Radius distance search query plan with and without index'}
  height={712}
  width={571}
/>
</div>

### Cypher Bounding Box Search

The next type of spatial search to consider is searching within a rectangular bounding box.

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/bounding_box.png'}
  alt={'Bounding box search'}
  height={341}
  width={477}
/>
</div>

We will use the [`point.withinBBox`](https://neo4j.com/docs/cypher-manual/current/functions/spatial/#functions-withinBBox) Cypher predicate function to accomplish this. This function takes 3 arguments: a point value that defines a location and two other point values that define the bounding box's lower left and upper right points.

```Cypher
MATCH (p:Point)
WHERE point.withinBBox(
  p.location,
  point({longitude:-122.325447, latitude: 37.55948 }),  // Lower left point
  point({longitude:-122.314675 , latitude: 37.563596})) // Upper right point
RETURN p
```

We can prepend `PROFILE` to the above Cypher query to verify that we are indeed using the index for this spatial search operation.

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/index2.png'}
  alt={'Query plan - using index for bounding box search'}
  height={686}
  width={362}
/>
</div>

### Point In Polygon Search

Point in polygon search will return points within the bounds of an arbitrary polygon. Point in polygon search is not natively supported in Neo4j, however with some help on the client side of our application we can accomplish a polygon search while taking advantage of the point database index to ensure the search is fast. To accomplish this we will use the following steps:

1. Convert polygon to bounding box
2. Cypher `withinBBox` query
3. Filter results to those within polygon

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/point_in_polygon_search.png'}
  alt={'Node chart visualization example'}
  height={478}
  width={551}
/>
</div>

Index backed point in polygon search can be accomplished by first converting the polygon to a bounding box, using Cypher's `point.withinBBox` predicate function to find points within the bounding box (using our database index), and then filtering the results on the client to the polygon bounds. [Example code here](https://github.com/johnymontana/geospatial-graph-demos/blob/main/src/index.html#L175-L231), using [Turf.js](https://turfjs.org/):

```js
const polygon = layer.toGeoJSON();
var bbox = turf.bbox(polygon); // convert polygon to bounding box

// Within Bounding Box Cypher query
const cypher = `
    MATCH (p:Point)-[:HAS_GEOMETRY]-(poi:PointOfInterest)-[:HAS_TAGS]->(t:Tags)
    WHERE point.withinBBox(
        p.location,
        point({longitude: $lowerLeftLon, latitude: $lowerLeftLat }),
        point({longitude: $upperRightLon, latitude: $upperRightLat}))
    RETURN p { latitude: p.location.latitude,
               longitude: p.location.longitude,
               name: poi.name,
               categories: labels(poi),
               tags: t{.*}
            } AS point
          `;

var session = driver.session({
    database: "osmpois",
    defaultAccessMode: neo4j.session.READ,
});

session
    .run(cypher, {
        lowerLeftLat: bbox[1],
        lowerLeftLon: bbox[0],
        upperRightLat: bbox[3],
        upperRightLon: bbox[2],
    })
    .then((result) => {
        const bboxpois = [];
        result.records.forEach((record) => {
            const poi = record.get("point");
            var point = [poi.longitude, poi.latitude];
            bboxpois.push(point);
        });
        // filter results of bouding box query to polygon bounds
        const poisWithin = turf.pointsWithinPolygon(
            turf.points(bboxpois),
            polygon
        );

        poisWithin.features.forEach((e) => {
            L.marker([
                e.geometry.coordinates[1],
                e.geometry.coordinates[0],
            ])
            .addTo(map)
            .bindPopup("Polygon");
        })
```


You can find the code for the spatial search examples (including using the Leaflet.js web map with Neo4j) [on GitHub: johnymontana/geospatial-graph-demos](https://github.com/johnymontana/geospatial-graph-demos)

### Working With Line Geometries

So far we've been working with points, let's take a quick look at an example using line geometries. For this quick example I exported my Strava activities data and loaded into Neo4j.

<div style={{  margin: 'auto' }}>
<Image
  src={'/static/images/nodes2022/bloom2.png'}
  alt={'Node chart visualization example'}
  height={693}
  width={800}
/>
</div>

To import data, first export user data from Strava then to add activities we will use the exported `activities.csv` file and load individual activity routes from the GPX files (first converted to geojson):

```Cypher
// Create Activity Nodes
LOAD CSV WITH HEADERS FROM "file:///activities.csv" AS row
MERGE (a:Activity {activity_id: row.`Activity ID`})
SET a.filename = row.Filename,
    a.activity_type = row.`Activity Type`,
    a.distance = toFloat(row.Distance),
    a.activity_name = row.`Activity Name`,
    a.activity_data = row.`Activity Date`,
    a.activity_description = row.`Activity Description`,
    a.max_grade = toFloat(row.`Max Grade`),
    a.elevation_high = toFloat(row.`Elevation High`),
    a.elevation_loss = toFloat(row.`Elevation Loss`),
    a.elevation_gain = toFloat(row.`Elevation Gain`),
    a.elevation_low = toFloat(row.`Elevation Low`),
    a.moving_time = toFloat(row.`Moving Time`),
    a.max_speed = toFloat(row.`Max Speed`),
    a.avg_grade = toFloat(row.`Average Grade`);

// Parse geojson geometries and create Geometry:Line nodes
MATCH (a:Activity)
WITH a WHERE a.filename IS NOT NULL AND a.filename CONTAINS ".gpx"
MERGE (n:Geometry {geom_id:a.activity_id })
MERGE (n)<-[:HAS_FEATURE]-(a)
WITH n,a
CALL apoc.load.json('file:///' + replace(a.filename, '.gpx', '.geojson')) YIELD value
UNWIND value.features[0].geometry.coordinates AS coord
WITH n, collect(point({latitude: coord[1], longitude: coord[0]})) AS coords
SET n.coordinates = coords
SET n:Line;
```

We store the routes as an array of point values to represent the line geometry of each route. Now, we can use the same spatial search techniques described above with our line geometries.

<Image
  src={'/static/images/nodes2022/line_search.png'}
  alt={'Airport graph model'}
  height={933}
  width={1102}
/>

By using the list predicate `any` Cypher function in our query we can check if at least one point in the line are within the radius distance search distance.

```Cypher
WITH point({latitude: $latitude, longitude: $longitude}) AS radiusCenter
MATCH (g:Geometry)<-[:HAS_FEATURE]-(a:Activity)
WHERE any(
  p IN g.coordinates WHERE point.distance(p, radiusCenter) < $radius
)
RETURN *
```

Code for this example is available [on GitHub here](https://github.com/johnymontana/geospatial-graph-demos#line-geometry-search).

## Routing With Graph Algorithms

One more use case I'd like to touch on besides spatial search is that of routing. We'll use the [Graph Data Science Neo4j Sandbox](https://dev.neo4j.com/sandbox) to find airline routes between airports. 

<Image
  src={'/static/images/nodes2022/routing1.png'}
  alt={'Airport graph model'}
  height={696}
  width={1301}
/>

There are a number of relevant path finding algorithms we might consider. In this case we will use Dijkstra's algorithm, an extension of breadth first search that uses relationship weights to find the shortest path between two nodes, or airports in our case. Dijkstra's algorithm is one of many available in Neo4j's Graph Data Science library. First, we will project our airline route graph into an in-memory projected graph structure used by the Graph Data Science library:

```Cypher
CALL gds.graph.project(
  'routes-weighted',
  'Airport',
  'HAS_ROUTE',
  {
    relationshipProperties: 'distance'
  }
) YIELD graphName, nodeProjection, nodeCount,
  relationshipProjection, relationshipCount
```

Then, to search for the shortest path between two airports we use the [`gds.shortestPath.dijkstra` procedure](https://neo4j.com/docs/graph-data-science/current/algorithms/dijkstra-source-target/):

```Cypher
MATCH (source:Airport {iata: 'PUQ'}), (target:Airport {iata: 'HNL'})
CALL gds.shortestPath.dijkstra.stream('routes-weighted', {
  sourceNode: source,
  targetNode: target,
  relationshipWeightProperty: 'distance'
})
YIELD index, sourceNode, targetNode,
   totalCost, nodeIds, costs, path
RETURN
nodes(path) as path
```

<Image
  src={'/static/images/nodes2022/routing2.png'}
  alt={'Routing between airports using the Dijkstra algorithm'}
  height={1202}
  width={1600}
/>


In this post we introduced some of the spatial functionality natively supported by Neo4j including the point type and related Cypher functions and demonstrated how to accomplish various spatial search operations as well as a brief look at routing with graph algorithms.



## Resources

- [Cypher manual for spatial Cypher functions](https://neo4j.com/docs/cypher-manual/current/syntax/spatial/)
- [Spatial search map example code](https://github.com/johnymontana/geospatial-graph-demos)
- [Daylight Earth Table](https://daylightmap.org/earth/)
- [Importing Daylight Earth Table points of interest into Neo4j (Python code)](https://github.com/johnymontana/daylight-earth-graph)
- [Spatial search Leaflet.js + Neo4j demo code](https://github.com/johnymontana/geospatial-graph-demos)

For comments and discussion please [join the conversation for this post at Dev.to](https://dev.to/lyonwj/spatial-search-functionality-with-neo4j-1c8h)


Radius Distance, Bounding Box, and Point in Polygon Search With Graphs

Spatial Search Functionality With Neo4j


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

In this post we explore building an interactive graph data visualization using GraphQL as our data source, with the [Lobsters GraphQL API we built previously.](https://lyonwj.com/blog/graphql-server-next-js-neo4j-aura-vercel) We'll be using Next.js, Vercel, and the `react-force-graph` library to add a graph visualization of users, tags, and articles posted to [Lobste.rs](https://lobste.rs/). Previously we saw how to use GitHub Actions and Neo4j Aura to automate the import of Lobsters data into Neo4j, so [check out the previous post](https://lyonwj.com/blog/no-cost-data-scraping-github-actions-neo4j-aura) if you're interested in that part.

<Subscribe />

We're building this application on the [Neo4j livestream](https://youtube.com/neo4j) so you can check out the video recording to accompany this blog post here:

<YouTube youTubeId="yLDO_FH6hGY" />

## Force Directed Layouts & Graph Visualization

In the [previous post](https://lyonwj.com/blog/graphql-server-next-js-neo4j-aura-vercel#interactive-graph-visualization-with-neo4j-bloom) we talked a bit about graph visualization with tools like Neo4j Bloom and Neo4j Browser, both of which are available in [Neo4j Aura](https://dev.neo4j.com/neo4j-aura). Both Neo4j Bloom and Neo4j Browser (as well as many other tools and libraries) use a force-directed layout for graph visualization.

In a force-directed layout the nodes are positioned according to a physics simulation where connected nodes are attracted (think of the relationship acting as a spring) and nodes that are not connected to each other are pushed away from each other (like electrons). Force directed layouts are useful for graph visualization because they result in visual clusters of connected nodes that can help interpret the structure of the graph at a glance.

<Image
  src={'/static/images/graphql-viz/forcedirected.png'}
  alt={'Node chart visualization example'}
  height={1054}
  width={1718}
/>

## Graph Data Visualization With `react-force-graph`

The [`force-graph`](https://github.com/vasturiano/force-graph) JavaScript library can be used to help build interactive data visualizations using a force-directed layout. It uses HTML5 Canvas for rendering and the `d3-force` layout algorithm. There are also 3D and AR/VR versions of the library, as well as a React flavor, which is what we'll be using.

### Using `react-force-graph` with Next.js

First, let's install the `react-force-graph-2d` package:

```shell
yarn add react-force-graph-2d
```

We're using Next.js to build our application which will by default use server side rendering (SSR) to render our pages, however we don't want our graph visualization to be rendered on the server because it depends on elements of client JavaScript that runs in the browser to render the visualization. So we'll use the dynamic import feature of Next.js and disable SSR.

We'll create a simple module that imports and exports the `ForceGraph2D` module:

```js
import ForceGraph2D from 'react-force-graph-2d';
export default ForceGraph2D;
```

Then import that module using a dynamic import:

```js
import dynamic from 'next/dynamic';

const NoSSRForceGraph = dynamic(() => import('../lib/NoSSRForceGraph'), {
  ssr: false
});
```

The force-graph component expects our graph data in a certain format. Specifically, we'll need to provide the node and relationship data in an object that contains two arrays, `nodes` and `links`. The `nodes` array will contain objects that represent the nodes in the visualization and must include at least an `id` value that identifies uniqueness of the node. The `links` array represents our relationships connecting nodes and reference their `source` and `target` nodes using the node id.

Let's hardcode some fake node and link data to get going with a very basic visualization:

```js
const myData = {
  nodes: [{ id: 'a' }, { id: 'b' }, { id: 'c' }],
  links: [
    { source: 'a', target: 'b' },
    { source: 'c', target: 'a' }
  ]
};

export default function Home() {
  return <NoSSRForceGraph graphData={myData} />;
}
```

Here's our basic graph visualization using `react-force-graph` and Next.js so far:

<Image
  src={'/static/images/graphql-viz/nodechart0.png'}
  alt={'Node chart visualization'}
  height={636}
  width={1244}
/>

Now it's time to bring in our data!

## Setting Up Apollo Client In Next.js

Since Next.js is a fullstack framework we'll be using GraphQL in two ways: 1) to implement our GraphQL server using the [API Routes](https://lyonwj.com/blog/graphql-server-next-js-neo4j-aura-vercel) feature of Next.js, and 2) as a client of the GraphQL API to fetch and consume data via GraphQL.

We set up the GraphQL server in the [previous episode](https://lyonwj.com/blog/graphql-server-next-js-neo4j-aura-vercel) so now it's time to set up the GraphQL client. We'll be using Apollo Client, which is one of the most popular GraphQL clients for React applications and will allow us to use some nice React Hooks for working with GraphQL data. First, let's install the `@apollo/client` package:

```shell
yarn add @apollo/client
```

Next, we need to create an Apollo Client instance and inject that client instance into the React component hierarchy using the `ApolloProvider` component so that the client instance will be available to all our React components. I covered setting up Apollo Client in a Next.js app in this [blog post](https://lyonwj.com/blog/grandstack-podcast-app-next-js-graphql-authentication) for the GRANDstack podcast application, so check that post out for a bit more detail, but we'll update `pages/_app.js` to create an Apollo Client instance:

```js
import {
  ApolloProvider,
  ApolloClient,
  InMemoryCache,
  HttpLink
} from '@apollo/client';

const createApolloClient = () => {
  const link = new HttpLink({
    uri: '/api/graphql'
  });

  return new ApolloClient({
    link,
    cache: new InMemoryCache()
  });
};

function MyApp({ Component, pageProps }) {
  return (
    <ApolloProvider client={createApolloClient()}>
      <Component {...pageProps} />)
    </ApolloProvider>
  );
}

export default MyApp;
```

## Pulling In GraphQL Data

Now we can use the Apollo Client React hooks in our app. Let's replace the placeholder data in our graph visualization with data populated from a GraphQL query of the Lobsters graph GraphQL API. We'll need to write a GraphQL query that will return the most recent articles (let's get the most recent 30), as well as the tags of the article and the user who submitted it.

```GraphQL
{
    articles(options: { limit: 30, sort: { created: DESC } }) {
      __typename
      id
      url
      title
      created
      tags {
        __typename
        name
      }
      user {
        username
        avatar
        __typename
      }
    }
  }
```

<Image
  src={'/static/images/graphql-viz/playground1.png'}
  alt={'GraphQL Playground'}
  height={1558}
  width={3182}
/>

Let's update index.js to use this GraphQL query to fetch data from our GraphQL endpoint using the `useQuery` hook. We'll also need to write a function (`formatData`) to convert the JSON data returned from our GraphQL query to the `nodes` and `links` format expected by the force-graph component. Finally, we'll save the result of that transformation into a React state variable (`graphData`) and pass that to the force-graph component. That way as we update the `graphData` state variable our visualization will update.

```js
import dynamic from 'next/dynamic';
import { useQuery, gql } from '@apollo/client';
import { useState } from 'react';

const NoSSRForceGraph = dynamic(() => import('../lib/NoSSRForceGraph'), {
  ssr: false
});

const mostRecentQuery = gql`
  {
    articles(options: { limit: 30, sort: { created: DESC } }) {
      __typename
      id
      url
      title
      created
      tags {
        __typename
        name
      }
      user {
        username
        avatar
        __typename
      }
    }
  }
`;

const formatData = (data) => {
  // this could be generalized but let's leave that for another time

  const nodes = [];
  const links = [];

  if (!data.articles) {
    return;
  }

  data.articles.forEach((a) => {
    nodes.push({
      id: a.id,
      url: a.url,
      __typename: a.__typename,
      title: a.title
    });

    links.push({
      source: a.user.username,
      target: a.id
    });

    a.tags.forEach((t) => {
      nodes.push({
        id: t.name,
        __typename: t.__typename
      });
      links.push({
        source: a.id,
        target: t.name
      });
    });

    nodes.push({
      id: a.user.username,
      avatar: a.user.avatar,
      __typename: a.user.__typename
    });
  });

  return {
    // nodes may be duplicated so use lodash's uniqBy to filter out duplicates
    nodes: _.uniqBy(nodes, 'id'),
    links
  };
};

export default function Home() {
  const [graphData, setGraphData] = useState({ nodes: [], links: [] });

  const { data } = useQuery(mostRecentQuery, {
    onCompleted: (data) => setGraphData(formatData(data))
  });

  return (
    <NoSSRForceGraph
      graphData={graphData}
      nodeLabel={(node) => {
        return node.id;
      }}
      nodeAutoColorBy={'__typename'}
      nodeRelSize={8}
    />
  );
}
```

We also set the node label in the visualization to be the `id` value and use the `nodeAutoColorBy` configuration to assign colors by `__typename` (which map to node labels in the database). Here's what our visualization looks like now:

<Image
  src={'/static/images/graphql-viz/nodechart1.png'}
  alt={'Node chart visualization'}
  height={1680}
  width={2194}
/>

## Adding More Nodes On Click

We want our visualization to be interactive and enable users to explore the graph. Let's add more data to the visualization when a user clicks on one of the tag nodes, allowing the user to find more articles for tags they might be interested in. To do this we'll write another GraphQL query (`moreArticlesQuery`) that will find the 10 most recent articles for a given tag. Then, we'll use the `useLazyQuery` hook from Apollo Client to execute this query when a user clicks on a tag node in the `onNodeClick` handler for the force-graph component. We'll also open the article in a new tab if the user clicks on an article node.

```js
import dynamic from 'next/dynamic';
import { useQuery, useLazyQuery, gql } from '@apollo/client';
import { useState } from 'react';

const NoSSRForceGraph = dynamic(() => import('../lib/NoSSRForceGraph'), {
  ssr: false
});

const moreArticlesQuery = gql`
  query articlesByTag($tag: String) {
    articles(
      where: { tags: { name: $tag } }
      options: { limit: 10, sort: { created: DESC } }
    ) {
      __typename
      id
      url
      title
      created
      tags {
        __typename
        name
      }
      user {
        username
        avatar
        __typename
      }
    }
  }
`;

export default function Home() {
  const [graphData, setGraphData] = useState({ nodes: [], links: [] });
  const { data } = useQuery(mostRecentQuery, {
    onCompleted: (data) => setGraphData(formatData(data))
  });
  const [loadMoreArticles, { called, loading, data: newData }] = useLazyQuery(
    moreArticlesQuery,
    {
      onCompleted: (data) => {
        const newSubgraph = formatData(data);
        setGraphData({
          nodes: _.uniqBy([...graphData.nodes, ...newSubgraph.nodes], 'id'),
          links: [...graphData.links, ...newSubgraph.links]
        });
      }
    }
  );

  return (
    <NoSSRForceGraph
      graphData={graphData}
      nodeLabel={(node) => {
        return node.id;
      }}
      nodeAutoColorBy={'__typename'}
      nodeRelSize={8}
      onNodeClick={(node, event) => {
        console.log('You clicked me!');
        console.log(node);

        if (node.__typename === 'Tag') {
          console.log('Lode more articles');
          loadMoreArticles({ variables: { tag: node.id } });
        } else if (node.__typename == 'Article') {
          window.open(node.url, '_blank');
        }
      }}
    />
  );
}
```

Now here's what our visualization looks like after expanding a few tags.

<Image
  src={'/static/images/graphql-viz/nodechart2.png'}
  alt={'Node chart visualization'}
  height={1900}
  width={1984}
/>

It's a bit difficult to see at a glance what the tags and article titles are. Let's use text to represent the nodes in our visualization so we can see the relevant information without hovering over the nodes to see their labels.

## Representing Nodes With Text

In order to change the visual representation of each node we'll need to override the `nodeCanvasObject` function. This function will allow us to the HTML Canvas API to style the nodes however we like. Using [this example](https://vasturiano.github.io/force-graph/example/text-nodes/) as a guide, let's use Canvas to style our nodes as text. We'll also use the user's avatar image to represent the user node.

```jsx
return (
  <NoSSRForceGraph
    {/* other props omitted*/}
    nodeCanvasObject={(node, ctx, globalScale) => {
        if (node.__typename === "Tag" || node.__typename === "Article") {
          const label = node.title || node.id;
          const fontSize = 16 / globalScale;
          ctx.font = `${fontSize}px Sans-Serif`;
          const textWidth = ctx.measureText(label).width;
          const bckgDimensions = [textWidth, fontSize].map(
            (n) => n + fontSize * 0.2
          );
          ctx.fillStyle = `rgba(255, 255, 255, 0.8)`;
          ctx.fillRect(
            node.x - bckgDimensions[0] / 2,
            node.y - bckgDimensions[1] / 2,
            ...bckgDimensions
          );
          ctx.textAlign = "center";
          ctx.textBaseline = "middle";
          ctx.fillStyle = node.color;
          ctx.fillText(label, node.x, node.y);

          node.__bckgDimensions = bckgDimensions;
        } else if (node.__typename === "User") {
          const size = 12;
          const img = new Image();
          img.src = node.avatar;
          ctx.drawImage(img, node.x - size / 2, node.y - size / 2, size, size);
        }
    }}
  />
)
```

And now we can see at a glance the article titles and tags:

<Image
  src={'/static/images/graphql-viz/textchart1.png'}
  alt={'Representing nodes with text'}
  height={1464}
  width={2922}
/>

That's all for now, in the next post we'll take a look at improving some of the styling of our visualization and bring in some other visualization elements to help us explore the data and find articles we're interested in. You can find the current version of the app [here](https://lobste-rs-graph.vercel.app) and the GraphQL API is available [here](https://lobste-rs-graph.vercel.app/api/graphql) if you'd like to try some queries against it.

## Resources

- [Code is available on GitHub](https://github.com/johnymontana/lobste.rs-graph)
- [Apollo Client `useLazyQuery` hook](https://www.apollographql.com/docs/react/api/react/hooks/#uselazyquery)
- [HTML Canvas API Tutorial](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial)
- [Lodash `uniqBy` array function](https://lodash.com/docs/4.17.15#uniqBy)
- [Dynamic import with Next.js](https://nextjs.org/docs/advanced-features/dynamic-import)
- [Building A GraphQL API With Next.js API Routes](https://lyonwj.com/blog/graphql-server-next-js-neo4j-aura-vercel)
- [Canvas drawImage](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage)
- [The `force-graph` package](https://github.com/vasturiano/force-graph)
- [The `react-force-graph` package](https://github.com/vasturiano/react-force-graph)

For comments and discussion please [join the conversation for this post at Dev.to](https://dev.to/lyonwj/graph-data-visualization-with-graphql-react-force-graph-18pk)


Building Interactive Graph Data Visualizations With GraphQL

Graph Data Visualization With GraphQL & react-force-graph


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

In the [previous post](https://lyonwj.com/blog/no-cost-data-scraping-github-actions-neo4j-aura) we started working with data from the Lobsters social news aggregator site in [Neo4j Aura](https://dev.neo4j.com/neo4j-aura) with the ultimate goal of building a fullstack application that allows us to explore the data as an interactive graph visualization. In this post we continue our journey toward that fullstack goal, focusing on getting our API layer up and running using Next.js API routes, the Neo4j GraphQL library, Vercel and Neo4j Aura.

<Subscribe />

We're building this application on the [Neo4j livestream](https://youtube.com/neo4j) so you can check out the video recording to accompany this blog post here:

<YouTube youTubeId="XVhdpP2GtPQ" />

## Interactive Graph Visualization With Neo4j Bloom

Our goal is to build a web application that will demonstrate the power of data visualization when using social network data to, in this case, help us discover interesting and relevant content. Before we dive into building our fullstack application it's worth exploring what sort of graph data visualization tooling is available to us out of the box with Neo4j.

Perhaps the most relevant tool we might consider is [Neo4j Bloom.](https://lyonwj.com/blog/election-night-dashboard-neo4j-charts-bloom-visualization) Bloom is included in Neo4j Aura and Neo4j Desktop and allows the user to explore the graph visually without writing Cypher. Bloom is a standalone application that enables users to search for patterns, filter, explore and share graph visualizations.

<Image
  src={'/static/images/graphql-server-nextjs/bloom1.png'}
  alt={'Neo4j Bloom graph visualization'}
  height={2036}
  width={3584}
/>

While Bloom is a powerful tool for use with Neo4j, it's not quite what we want for this project as we want to build a most custom and bespoke experience. There are also other "graph apps" available in Neo4j Desktop, such as the Charts app and Neomap that we can use to build visualizations and dashboards. I covered these in a previous post: ["Building An Election Night Dashboard With Neo4j Graph Apps: Bloom, Charts, & Neomap".](https://lyonwj.com/blog/election-night-dashboard-neo4j-charts-bloom-visualization)


## Next.js 

[Next.js](https://nextjs.org/) is a fullstack React framework built and maintained by [Vercel](https://vercel.com/). Next.js includes many features out of the box that we typically need to set up in React applications - things like file-system routing, server side rendering, API routes, etc - which means we can focus on building our application and not boilerplate setup and configuration. 

We covered Next.js in [a previous blog post](https://lyonwj.com/blog/grandstack-podcast-app-next-js-graphql-authentication) so I won't go over all the features now, but I'm a big fan of Next.js and use it with most of my new projects now.

<Image
  src={'/static/images/graphql-server-nextjs/nextjs1.png'}
  alt={'Next.js features'}
  height={1684}
  width={2016}
/>

### `create-next-app`

The easiest way to get started with Next.js is to use [the `create-next-app` CLI](https://nextjs.org/docs/api-reference/create-next-app). This is a command line tool that enables us to quickly start building a new Next.js application. We can use it to create a new skeleton Next.js project or select from many of the example Next.js projects.

Let's use this to start a new Next.js application in our Lobsters Graph repository:

```shell
npx create-next-app next
```

We can now navigate to the `next` directory, and run `yarn dev` to start a local web server serving our Next.js application. We should see something like this with some placeholder content:

<Image
  src={'/static/images/graphql-server-nextjs/nextjs2.png'}
  alt={'Next.js features'}
  height={1350}
  width={2422}
/>

In this post we're going to focus on building the GraphQL API for our application, rather than the frontend so we won't cover anything React specific today. Instead, we'll be using Next.js' API Routes feature to build our GraphQL API.

## Next.js API Routes

Next.js has support for [creating API endpoints](https://nextjs.org/docs/api-routes/introduction) to add backend functionality to our Next.js application - it really is a fullstack framework after all. To create a new API route we just create a new file in `pages/api` that will be mapped to a new API endpoint.

The skeleton Next.js application we created with `create-next-app` includes an API route example in `pages/api/hello.js`:

```js
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction

export default function handler(req, res) {
  res.status(200).json({ name: 'John Doe' })
}
```

If we make a request to `localhost:3000/api/hello` we'll get back a simple JSON object:

```json
{
    name: "John Doe"
}

```

Let's use this API route functionality to add a GraphQL endpoint to our Next.js application.

## Creating A GraphQL Server In A Next.js API Route

Following [the GraphQL example linked in the Next.js documentation](https://github.com/vercel/next.js/tree/canary/examples/api-routes-graphql), we'll use the `micro` package and `apollo-server-micro` to set up a simple GraphQL server as an API route.

First, we'll install the necessary dependencies:

```shell
yarn add apollo-server-micro micro graphql
```

Micro is an HTTP server that works well with Next.js and more importantly for our purposes has an Apollo Server implementation. To create a GraphQL server with Apollo Server we need to create two things: GraphQL type definitions that define the data available in the API, and GraphQL resolver functions that contain the logic for actually resolving GraphQL operations. Apollo Server takes these two inputs, combines them into an executable GraphQL schema and handles the HTTP network layer involved in serving a GraphQL API.

Let's create simple GraphQL type definitions and a single resolver function to get our API up and running as an API route:

```js
import { gql, ApolloServer } from "apollo-server-micro";

const typeDefs = gql`
  type User {
    id: ID
  }

  type Query {
    getUser: User
  }
`;

const resolvers = {
  Query: {
    getUser: () => {
      return {
        id: "Foo",
      };
    },
  },
};

const apolloServer = new ApolloServer({
  typeDefs,
  resolvers,
});

const startServer = apolloServer.start();

export default async function handler(req, res) {

  await startServer;
  await apolloServer.createHandler({
    path: "/api/graphql",
  })(req, res);
}

export const config = {
  api: {
    bodyParser: false,
  },
};
```

We define a single Query field `getUser` and a single `User` type that has only one field, `id` and a resolver function that returns a single hardcoded `User` object.

## Using GraphQL Playground With Apollo Server v3

In previous versions of Apollo Server by default the GraphQL Playground in-browser tool for exploring GraphQL APIs was available. However, GraphQL Playground has been deprecated for some time now and the latest release of Apollo Server, v3, instead links to the hosted Apollo Studio tool on the "landing page" of the GraphQL API (the page loaded when the GraphQL endpoint is loaded in a web browser).

Apollo Studio is great, but since we want this to be a public GraphQL API that anyone can explore I want GraphQL Playground to be served on the landing page. Fortunately, we can [enable GraphQL Playground as a plugin](https://www.apollographql.com/docs/apollo-server/migration/#graphql-playground) with Apollo Server 3 with these changes:

```js
import { ApolloServerPluginLandingPageGraphQLPlayground } from "apollo-server-core";

...

const apolloServer = new ApolloServer({
  typeDefs,
  resolvers,
  playground: true,
  plugins: [ApolloServerPluginLandingPageGraphQLPlayground()],
});
```

Now when we load `localhost:3000/graphql/api` in a web browser we should see the familiar GraphQL Playground tool. To verify that our GraphQL server is working properly we can run this query:

```graphql
{
    getUser {
        id
    }
}
```

And we should see the simple result returned by our `getUser` resolver function:

```json

{
    data: {
        getUser: {
            id: "Foo"
        }
    }
}

```

Now let's update our GraphQL schema from the placeholder to one that models our Lobsters graph data and works with Neo4j.


## Using The Neo4j GraphQL Library

The [Neo4j GraphQL library](https://dev.neo4j.com/graphql) allows us to build Node.js GraphQL APIs backed by Neo4j without writing any resolvers. All we need to do is write GraphQL type definition that define the data model of our database and the Neo4j GraphQL library takes care of the rest - generating a full CRUD GraphQL API and resolvers and translating arbitrary GraphQL operations to database queries.


<Image
  src={'/static/images/graphql-server-nextjs/neo4jgraphql1.png'}
  alt={'The Neo4j GraphQL library'}
  height={930}
  width={2544}
/>

First, let's install a couple of additional dependencies, the Neo4j GraphQL library and the Neo4j JavaScript driver:

```shell
yarn add @neo4j/graphql neo4j-driver
```


Next, we'll need to create the GraphQL type definitions that map to the property graph model we're using for the Lobsters data. If we refer back to the [previous post](https://lyonwj.com/blog/no-cost-data-scraping-github-actions-neo4j-aura) we can use the graph data model diagram we created using the Arrows.app tool:

<Image
  src={'/static/images/data-scraping/datamodel.png'}
  alt={'Our data model using Arrows.app'}
  height={1354}
  width={1858}
/>

The Neo4j GraphQL library uses the following conventions to map GraphQL type definitions to the property graph model:

* GraphQL types map to node labels in the property graph model
* GraphQL scalar fields map to node properties in the property graph model
* GraphQL object and object array fields map to relationships in the property graph model
* The `@relationship` directive is used in the GraphQL type definitions to encode the relationship type and direction in the property graph model

Applying these conventions we end up with the following GraphQL type definitions that map to our Lobsters property graph in Neo4j:

```graphql
type User {
    username: String
    created: DateTime
    karma: Int
    about: String
    avatar: String
    articles: [Article] @relationship(type: "SUBMITTED", direction: OUT)
    invited: [User] @relationship(type: "INVITED_BY", direction: IN)
    invited_by: [User] @relationship(type: "INVITED_BY", direction: OUT)
  }

  type Article {
      id: ID
      url: String
      score: Int
      title: String
      comments: String
      created: DateTime
      user: User @relationship(type: "SUBMITTED", direction: IN)
      tags: [Tag] @relationship(type: "HAS_TAG", direction: OUT)
  }

  type Tag {
      name: String
      articles: [Article] @relationship(type: "HAS_TAG", direction: IN)
  }
```

Now we'll remove the resolver functions from our placeholder GraphQL schema since we don't need to write manual resolvers when using the Neo4j GraphQL library and replace our GraphQL type definitions with the ones we wrote above.

We'll also create a Neo4j JavaScript driver instance to connect to our Neo4j Aura database, using environment variables for the connection credentials and we'll pass our GraphQL type definitions to the `Neo4jGraphQL` class constructor to generate our GraphQL API.

We also make use of the [`@exclude` directive](https://neo4j.com/docs/graphql-manual/current/) in our GraphQL type definitions to prevent any mutations from being added to the schema - we want this to be a read-only API, at least for now. 


```js

import { gql, ApolloServer } from "apollo-server-micro";
import { ApolloServerPluginLandingPageGraphQLPlayground } from "apollo-server-core";
import {Neo4jGraphQL} from "@neo4j/graphql"
import neo4j from "neo4j-driver"
import 'ts-tiny-invariant' // importing this module as a workaround for issue described here: https://github.com/vercel/vercel/discussions/5846


const typeDefs = gql`
  type User @exclude(operations: [CREATE, UPDATE, DELETE]) {
    username: String
    created: DateTime
    karma: Int
    about: String
    avatar: String
    articles: [Article] @relationship(type: "SUBMITTED", direction: OUT)
    invited: [User] @relationship(type: "INVITED_BY", direction: IN)
    invited_by: [User] @relationship(type: "INVITED_BY", direction: OUT)
  }

  type Article @exclude(operations: [CREATE, UPDATE, DELETE]) {
      id: ID
      url: String
      score: Int
      title: String
      comments: String
      created: DateTime
      user: User @relationship(type: "SUBMITTED", direction: IN)
      tags: [Tag] @relationship(type: "HAS_TAG", direction: OUT)
  }

  type Tag @exclude(operations: [CREATE, UPDATE, DELETE]) {
      name: String
      articles: [Article] @relationship(type: "HAS_TAG", direction: IN)
  }
`;

const driver = neo4j.driver(
    process.env.NEO4J_URI,
    neo4j.auth.basic(process.env.NEO4J_USER, process.env.NEO4J_PASSWORD)
)

const neoSchema = new Neo4jGraphQL({typeDefs, driver})

const apolloServer = new ApolloServer({
  schema: neoSchema.schema,
  playground: true,
  introspection: true,
  plugins: [ApolloServerPluginLandingPageGraphQLPlayground()],
});

const startServer = apolloServer.start();

export default async function handler(req, res) {
  await startServer;
  await apolloServer.createHandler({
    path: "/api/graphql",
  })(req, res);
}

export const config = {
  api: {
    bodyParser: false,
  },
};
```

Next.js supports setting environment variables using `.env` files, so let's create a `.env.local` file where we'll add the credentials for our Neo4j Aura instance. We'll also set the `DEBUG` environment variable to [enable debug logging for the Neo4j GraphQL library](https://neo4j.com/docs/graphql-manual/current/troubleshooting/). This will log the generated Cypher queries among other things, which can be helpful to understand how the database queries are generated from GraphQL operations.

```shell
NEO4J_USER=neo4j
NEO4J_URI=neo4j+s://YOUR NEO4J AURA URI HERE
NEO4J_PASSWORD=YOUR NEO4J AURA PASSWORD HERE
DEBUG=@neo4j/graphql:*
```

We now have a GraphQL endpoint running locally at `localhost:3000/api/graphql` that we can use to fetch data from our Neo4j Aura database using GraphQL. Next, we'll deploy our Next.js application on [Vercel](https://vercel.com/) so that it will be publicly accessible.


## Deploying To Vercel

Vercel is a cloud platform that we'll use to build and deploy our Next.js application. The frontend React application (once we build it!) will be built and hosted on a CDN and our GraphQL API route will be automatically deployed as a serverless function.

Vercel integrates with GitHub so once we commit our changes we can add our Next.js application by selecting the GitHub repository in Vercel. Because we're using a bit of a monorepo setup and our Next.js application is not in the root directory of the repository we just need to tell Vercel that the root directory of our Next.js application is the `next` directory. We'll also add our Neo4j Aura connection credentials as environment variables, setting values for `NEO4J_PASSWORD`, `NEO4J_URI`, and `NEO4J_USER` in the Vercel project configuration.

<Image
  src={'/static/images/graphql-server-nextjs/vercel1.png'}
  alt={'Adding a new project to Vercel'}
  height={2070}
  width={2752}
/>

Once we've added our project the Vercel build service will pull down our code from GitHub, build the project and deploy our Next.js application (static content to a CDN and our GraphQL API to a serverless function). Our project is automatically assigned a domain and SSL certificate! Because we connected our project via GitHub any commits and pull requests will trigger another build. Each build is assigned its own unique URL which means pull requests will be built and deployed as a "preview build" which we can test and share before deploying to our main domain. This is really great collaboration feature.

<Image
  src={'/static/images/graphql-server-nextjs/vercel2.png'}
  alt={'Vercel dashboard'}
  height={1408}
  width={2408}
/>

Since we added the GraphQL Playground plugin we can navigate to our Vercel project's URL in the browser [`https://lobste-rs-graph.vercel.app/api/graphql`](https://lobste-rs-graph.vercel.app/api/graphql) and test our GraphQL endpoint:

<Image
  src={'/static/images/graphql-server-nextjs/playground1.png'}
  alt={'GraphQL Playground and Vercel'}
  height={1780}
  width={3290}
/>

Now that we've got our GraphQL API up, running, and deployed on Vercel in the next post we'll start building out the frontend application, taking a look at graph data visualization in React with GraphQL.

## Resources

- [Code available on GitHub](https://github.com/johnymontana/lobste.rs-graph)
- [No Cost Data Scraping With GitHub Actions](https://lyonwj.com/blog/no-cost-data-scraping-github-actions-neo4j-aura)
- [Neo4j Aura](https://dev.neo4j.com/neo4j-aura)
- [Graph Data Visualization With Neo4j Bloom](https://dev.neo4j.com/bloom)
- [Next.js](https://nextjs.org)
- [Next.js API Routes](https://nextjs.org/docs/api-routes/introduction)
- [Neo4j GraphQL Library](https://dev.neo4j.com/graphql)
- [Using GraphQL Playground With Apollo Server v3](https://www.apollographql.com/docs/apollo-server/migration/#graphql-playground)
- [Vercel](https://vercel.com/)


For comments and discussion please [join the conversation for this post at Dev.to](https://dev.to/lyonwj/building-a-graphql-server-with-next-js-api-routes-g2)


Using Next.js API Routes And Deploying To Vercel & Neo4j Aura

Building A GraphQL Server With Next.js


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

When working with data a common task is fetching data from some external source on a recurring basis and importing into a database for further analysis or as part of our application. Setting up servers to handle this can be time consuming and error prone. I recently came across a workflow using [GitHub Actions](https://github.com/features/actions) and [Neo4j Aura](https://dev.neo4j.com/neo4j-aura) that makes this a breeze and with the free tiers of both GitHub Actions and Neo4j Aura is free to set up and run forever - great for side projects!

<Subscribe />

In this post we'll take a look at setting up this workflow to scrape data from the [Lobsters](https://lobste.rs/) news aggregator and import into a [Neo4j Aura Free](https://dev.neo4j.com/aura-landing) instance using GitHub Actions. We built this on the [Neo4j livestream](https://twitch.tv/neo4j) so check out the recording if you prefer video:

<YouTube youTubeId="jAweyWeO2cM" />

## A Look At The Data - Lobsters

[Lobsters](https://lobste.rs/) is a social news aggregator. Users post links to articles and the community votes and comments on them in a discussion thread. An algorithm determines the ranking of the submissions, with the most recent and noteworthy articles floating to the front page. To be able to submit and comment users must be invited by an existing user and this user-invite graph is publicly available, helping to keep discussions civil and avoid fraudulent upvoting to game submission rankings.

<Image
  src={'/static/images/data-scraping/lobsters.png'}
  alt={'The Lobste.rs Social News Aggregator'}
  height={1904}
  width={1902}
/>

I've been thinking about social networks recently and want to build an application to help explore relevant news articles using graph visualization so building something using data from Lobsters seems like a great fit. We want to import data about users and article submissions into Neo4j as the basis of our application, but how to get started? Fortunately, Lobsters makes two JSON endpoints available to fetch data about the [newest](https://lobste.rs/newest.json) and the ["hottest"](https://lobste.rs/hottest.json) submissions. Each has a similar format and looks like this.

<Image
  src={'/static/images/data-scraping/lobsters-json.png'}
  alt={'JSON data representing the latest stories submitted to Lobsters'}
  height={1782}
  width={1628}
/>

Now that we know what our JSON data looks like we can think about how to start importing this data into Neo4j Aura as a graph.

## Neo4j Aura

[Neo4j Aura](https://dev.neo4j.com/neo4j-aura) is a managed cloud service from Neo4j allowing us to run Neo4j clusters in the cloud with a few clicks and handles things like backups, upgrades, and administration so we don't have to worry about it. Best of all for our purposes, Neo4j Aura has a free tier that will allow us to build this project without even putting in a credit card.

Let's [sign-in to Neo4j Aura](https://dev.neo4j.com/aura) and create an Aura Free instance:

<Image
  src={'/static/images/data-scraping/aura1.png'}
  alt={'Provisioning a Neo4j Aura Free instance'}
  height={1924}
  width={2282}
/>

We'll immediately be given a generated password that we'll need to save to access our Neo4j Aura instance. We can change this later.

<Image
  src={'/static/images/data-scraping/aura2.png'}
  alt={'Be sure to save your password, you will need this later'}
  height={1114}
  width={1332}
/>

Next, our Aura instance will take a few moments to be provisioned.

<Image
  src={'/static/images/data-scraping/aura3.png'}
  alt={'Neo4j Aura instance is starting up...'}
  height={478}
  width={2318}
/>

Once our Neo4j Aura instance is ready we'll see the connection string in the dashboard. Let's open up Neo4j Browser to start working with our data.

<Image
  src={'/static/images/data-scraping/aura4.png'}
  alt={'Our Neo4j Aura instance is ready'}
  height={418}
  width={2286}
/>

Since we're working with JSON data we can make use of the [`apoc.load.json`](https://neo4j.com/labs/apoc/4.1/import/load-json/) procedure from the [APOC standard library](https://neo4j.com/labs/apoc/) to import a JSON file using Cypher. First, let's just pull this file in to make sure we can parse it. This Cypher statement will parse the newest Lobsters submissions and return an array of objects that we can work with in Cypher:

```cypher
CALL apoc.load.json("https://lobste.rs/newest.json") YIELD value
RETURN value
```

If we run this in Neo4j Browser we'll see the parsed array of objects returned. We haven't actually created any data in the database yet - we're just parsing the JSON file and returning the results.

<Image
  src={'/static/images/data-scraping/apoc1.png'}
  alt={'Using apoc.load.json'}
  height={1144}
  width={2178}
/>

We'll use Cypher to define the graph structure we want to create from this data, but we first need to think a bit about how we want to model this data as a graph. I think of graph modeling as a multi-step process, at the highest level the steps are:

1. Identify the entities in our data - these become nodes.
1. Identify how these entities are connected - these connections become relationships.
1. Identify the pieces of data that describe the entities or connections - these become properties.

[Arrows.app](https://arrows.app) is a great tool for drawing and storing these graph models. Here's what our graph model for the Lobsters data looks like in Arrows:

<Image
  src={'/static/images/data-scraping/datamodel.png'}
  alt={'Our data model using Arrows.app'}
  height={1354}
  width={1858}
/>

Now we can write the Cypher to import our data according to this model. We'll use `apoc.load.json` to parse the JSON file, then use `UNWIND` to iterate over this array of objects. We'll then use the `MERGE` Cypher clause to add users, articles, and tags to the database. The `MERGE` statement allows us to avoid creating duplicates in the graph - with `MERGE` only patterns that don't already exist in the graph are created.

This makes our import statement idempotent - we can run it over and over again on the same data and we won't make any changes to the database unless the data changes.

```cypher
CALL apoc.load.json("https://lobste.rs/newest.json") YIELD value
UNWIND value AS article
MERGE (s:User {username: article.submitter_user.username})
ON CREATE SET s.about = article.submitter_user.about,
              s.created = DateTime(article.submitter_user.created_at),
              s.karma = article.submitter_user.karma,
              s.avatar_url = "https://lobsete.rs" + article.submitter_user.avatar_url
MERGE (i:User {username: article.submitter_user.invited_by_user})
MERGE (i)<-[:INVITED_BY]-(s)
MERGE (a:Article {short_id: article.short_id})
SET a.url = article.url,
    a.score = article.score,
    a.created = DateTime(article.created_at),
    a.title = article.title,
    a.comments = article.comments_url
MERGE (s)-[:SUBMITTED]->(a)
WITH article, a
UNWIND article.tags AS tag
MERGE (t:Tag {name: tag})
MERGE (a)-[:HAS_TAG]->(t)
```

To verify we've imported this data correctly we can visualize it in Neo4j Browser - let's check to make sure the property values are what we expect and makes sense.

<Image
  src={'/static/images/data-scraping/apoc2.png'}
  alt={'Using apoc.load.json'}
  height={1040}
  width={1326}
/>

That's great, but the Lobsters data will be continually changing as new articles are submitted and voted on - we want to import it on an ongoing basis. Let's use GitHub Actions to do this!

## GitHub Actions And Flat Graph

[GitHub Actions](https://github.com/features/actions) allows us to define workflows that are triggered by GitHub events (like commits, pull requests, etc). We can write custom code to run when an Action is triggered or choose from existing Actions published by community members in the GitHub Action Marketplace. While commonly used for CI/CD workflows, we can also schedule Actions to run at a specified interval.

The GitHub team recently released the [Flat Data project](https://octo.github.com/projects/flat-data) which aims to simplify data and ETL workflows. Flat Data includes the Flat Data GitHub Action, a VSCode extension for creating Flat Data workflows, and a web app for viewing data. The Flat Data Action allows us to schedule a GitHub action to periodically fetch data via a URL or SQL statement, check the data into git, and run a postprocessing step to, for example, insert data into a database.

This sounds perfect for what we want to accomplish with Lobsters and Neo4j Aura - let's give it a try! First, we'll sign into GitHub and create a new repository:

<Image
  src={'/static/images/data-scraping/github1.png'}
  alt={'Creating a new GitHub repo'}
  height={1832}
  width={1936}
/>

Because we selected "Add a README file" we can immediately start to make commits to our repo and use the built-in editor in the GitHub web UI to set up our Flat Data Action. We can also clone the repo locally, but I'll just use the built-in text editor.

<Image
  src={'/static/images/data-scraping/github2.png'}
  alt={'Creating a new GitHub repo'}
  height={1080}
  width={3192}
/>

To create a new GitHub action we'll add a new file `.github/workflows/lobsters.yml`. In this YAML file we will

1. Give a name to our new Action - let's call it "Lobsters Data Import".
1. Identity the events we want to trigger this action. We want it to run anytime this YAML file is updated and also schedule it to run every 60 minutes.
1. We then define the steps we want to run in our Action:

  - First, we'll check out the repo using the `actions/checkout@v2` action
  - Then we'll use the `githubocto/flat@v2` Action to fetch our Lobsters JSON file and check it into our repo as `newest.json`

```yml
name: Lobsters Data Import

on:
  push:
    paths:
      - .github/workflows/lobsters.yml
  workflow_dispatch:
  schedule:
    - cron: '*/60 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Fetch newest
        uses: githubocto/flat@v2
        with:
          http_url: https://lobste.rs/newest.json
          downloaded_filename: newest.json
```

Using the in-browser editor in GitHub we should see something like this:

<Image
  src={'/static/images/data-scraping/github3.png'}
  alt={'Creating a new GitHub repo'}
  height={1924}
  width={2692}
/>

Our GitHub Action will run immediately after committing the new file and in a few seconds we'll see a new commit, checking in the `newest.json` file with the data from Lobsters.

<Image
  src={'/static/images/data-scraping/github4.png'}
  alt={'Creating a new GitHub repo'}
  height={496}
  width={2268}
/>

Great - we've fetched the newest Lobsters article submissions and checked them into our repository and scheduled a GitHub Action to refresh this data every hour. Now we're ready to update our GitHub Action to import data into Neo4j.

First, we'll need to add secrets to our GitHub repository so our Action can connect to our Neo4j Aura instance and we don't have to expose our connection credentials. In GitHub select the "Settings" tab then navigate to the "Secrets" section and select the "New repository secret" button.

<Image
  src={'/static/images/data-scraping/github5.png'}
  alt={'Creating a new GitHub repo'}
  height={928}
  width={1958}
/>

We'll create three secrets `NEO4J_USER`, `NEO4J_PASSWORD`, and `NEO4J_URI` with the connection credentials specific to the Neo4j Aura instance we created earlier.

<Image
  src={'/static/images/data-scraping/github6.png'}
  alt={'Creating a new GitHub repo'}
  height={532}
  width={1852}
/>

We'll now be able to reference these secrets values in the YAML file where we define our GitHub Action to connect to our Neo4j Aura instance.

### The Flat Graph GitHub Action

The Flat Data GitHub Action includes support for a post-processing step that allows us to run a JavaScript or Python script after the data is fetched. We could write a simple script to use the Neo4j language driver to connect to our Neo4j Aura instance and run a Cypher import statement, passing in the JSON data as a Cypher parameter, however there is also the [Flat *Graph*](https://github.com/marketplace/actions/flat-graph) GitHub Action which does just this.


<Image
  src={'/static/images/data-scraping/flat-graph.png'}
  alt={'Flat Graph GitHub Action'}
  height={1714}
  width={2862}
/>

Flat Graph is designed to work with the Flat Data GitHub Action and allows us to declare the Cypher import statement we want to run and our Neo4j Aura connection credentials as another step in our GitHub Action. Flat Graph will load the specified JSON file and pass it as a Cypher parameter called `$value` so our Cypher import statement just needs to reference this Cypher parameter to work with the data fetched by Flat Data in the previous step of our Action.

We'll use the Cypher statement we wrote above using `apoc.load.json`, but adapt it to use this convention. We'll also reference the Neo4j Aura connection credentials we defined as GitHub secrets. Let's update our `lobsters.yml` file:

```yml
name: Lobsters Data Import

on:
  push:
    paths:
      - .github/workflows/lobsters.yml
  workflow_dispatch:
  schedule:
    - cron: '*/60 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Fetch newest
        uses: githubocto/flat@v2
        with:
          http_url: https://lobste.rs/newest.json
          downloaded_filename: newest.json
      - name: Neo4j import
        uses: johnymontana/flat-graph@v1.2
        with:
          neo4j-user: ${{secrets.NEO4J_USER}}
          neo4j-password: ${{secrets.NEO4J_PASSWORD}}
          neo4j-uri: ${{secrets.NEO4J_URI}}
          filename: newest.json
          cypher-query: >
            UNWIND $value AS article
            MERGE (s:User {username: article.submitter_user.username})
            ON CREATE SET s.about = article.submitter_user.about,
              s.created = DateTime(article.submitter_user.created_at),
              s.karma = article.submitter_user.karma,
              s.avatar_url = "https://lobsete.rs" + article.submitter_user.avatar_url
            MERGE (i:User {username: article.submitter_user.invited_by_user})
            MERGE (i)<-[:INVITED_BY]-(s)
            MERGE (a:Article {short_id: article.short_id})
            SET a.url = article.url,
                a.score = article.score,
                a.created = DateTime(article.created_at),
                a.title = article.title,
                a.comments = article.comments_url
            MERGE (s)-[:SUBMITTED]->(a)
            WITH article, a
            UNWIND article.tags AS tag
            MERGE (t:Tag {name: tag})
            MERGE (a)-[:HAS_TAG]->(t)
```

Once we commit the change our Action will run - loading the latest Lobsters article data into our Neo4j Aura instance. This Action will run each hour updating our database with the latest Lobsters data.

## What's Next?

We've now set up a serverless data scraping workflow to fetch all new articles submitted to Lobsters and import into Neo4j Aura using GitHub Actions. Next, we'll take a look at data visualization options and we begin to explore this dataset and build a web application to help us find relevant information. If you'd like to follow along we'll be coding this on the Neo4j Livestream on [Twitch](https://twitch.tv/neo4j) and [YouTube](https://www.youtube.com/neo4j). You can also [subscribe to my newsletter](https://lyonwj.com/newsletter) to be notified as new posts are up. Happy graphing!

## Resources

- [Neo4j Aura](https://dev.neo4j.com/neo4j-aura)
- [Lobste.rs](http://lobste.rs/)
- [GitHub Actions](https://github.com/features/actions)
- [GitHub Flat Data](https://octo.github.com/projects/flat-data)
- [Flat Graph GitHub Action](https://github.com/marketplace/actions/flat-graph)
- [Code on GitHub](https://github.com/johnymontana/lobste.rs-graph)

For comments and discussion please [join the conversation for this post at Dev.to](https://dev.to/lyonwj/no-cost-data-scraping-with-github-actions-and-neo4j-aura-372b)


Using The Flat Data GitHub Action To Import Data From The Lobste.rs News Site

No Cost Data Scraping With GitHub Actions And Neo4j Aura

Building A GRANDstack Podcast App: Episode 6

Adding Frontend Features With Next.js, Chakra UI, & Apollo Client

Building A GRANDstack Podcast App: Episode 5

Building A Responsive Navigation Bar And Podcast Episode Feed With The Chakra UI React Component Library


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

*This is the fourth post in a series about building a podcast application using [GRANDstack](https://grandstack.io). Check out the previous episodes here:*
> * [**Episode 1:** "Podcast Search GraphQL API With Neo4j And The Podcast Index"](https://lyonwj.com/blog/grandstack-podcast-app-podcast-search-graphql-api)
> * [**Episode 2:** "GRANDcast.FM: User Auth & Podcast Subscribe Functionality"](https://lyonwj.com/blog/grandstack-podcast-app-user-auth-podcast-subscribe)
> * [**Episode 3:** "Parsing And Importing XML With Neo4j: Adding Episodes and Playlists"](https://lyonwj.com/blog/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists)

<Subscribe />


In this post we turn our attention to the frontend of the podcast application we've been building on the [Neo4j livestream](https://www.twitch.tv/neo4j). So far we’ve focused on the backend and GraphQL API layer. This week we got started on the front end, using the Next.js React framework. You can find the livestream recording embedded below or on the [Neo4j Youtube channel](https://youtube.com/neo4j).

<YouTube youTubeId="pB4YZBJmMl8" />

In this post we'll focus on getting started with Next.js, fetching data using GraphQL in our Next.js application, and setting up client-side authentication with GraphQL in Next.js so we can sign in and make authenticated requests to our GraphQL API.

# What Is Next.js?

Next.js is a web framework built on top of React that adds many features and conventions that aren't available by default with React. Things like server side rendering, file based routing, image optimization, code splitting and bundling, and adding server-side API routes are things we eventually need to deal with as our React application becomes more complex. With Next.js we have all these things (and lots more) available out of the box. I like to think of Next.js as React with batteries included.

<Image
  src="/static/images/grandcast-4/nextjs.png"
  alt="Next.js landing page"
  height={1636}
  width={2508}
/>

# Getting Started With Next.js

The easiest way to get started with Next.js is using the `create-next-app` CLI. Similar to `create-react-app` or [`create-grandstack-app`](https://grandstack.io/docs/getting-started-grand-stack-starter), we can create an initial Next.js application from the command line.

```shell
npx create-next-app next-app
```

The `create-next-app` command can take an example argument to specify an example template to use. There are two example Next.js templates for using Next.js with Neo4j:

* [with-neo4j](https://github.com/vercel/next.js/tree/canary/examples/with-neo4j)
* [with-apollo-neo4j-graphql](https://github.com/vercel/next.js/tree/canary/examples/with-apollo-neo4j-graphql)

We're going to skip these Neo4j specific templates since we've already built out our GraphQL API. In a future post we'll see how we can move our GraphQL API into Next.js, taking advantage of the serverless function deployment built into Next.js API routes.

To start our Next.js application we'll run 

```shell
npm run dev
```

which will start a local server at `localhost:3000` serving our Next.js application.


# GraphQL Data Fetching With Next.js

The first thing we want to do is make data fetching queries with GraphQL and display podcast data in our Next.js application. To do this we'll use Apollo Client. First, let's install Apollo Client, by default this will include the React hooks integration for Apollo Client as well.

```shell
npm i @apollo/client
```

In React applications we make use of the Provider pattern and the React Context API to make data available throughout the React component hierarchy. The React integration for Apollo Client includes an `ApolloProvider` component that we can use to inject an Apollo Client instance into the React component hierarchy, which will make Apollo Client available to any component in our application. To do this we need to create an Apollo Client instance and wrap our application's root component in the `ApolloProvider` component. In Next.js we can do this in the `_app.js` file.

```js
import '../styles/globals.css'
import {
  ApolloProvider,
  ApolloClient,
  InMemoryCache,
  HttpLink,
} from '@apollo/client'

function createApolloClient() {
  const link = new HttpLink({
    uri: 'http://localhost:4001/graphql',
  })

  return new ApolloClient({
    link,
    cache: new InMemoryCache(),
  })
}

function MyApp({ Component, pageProps }) {
  return (
    <ApolloProvider client={createApolloClient()}>
      <Component {...pageProps} />
    </ApolloProvider>
  )
}

export default MyApp
```

Note the use of an `HttpLink` instance. With Apollo Client, [links](https://www.apollographql.com/docs/link/) are composable units, similar to middleware, that allow us to inject logic into the networking layer of a GraphQL request. Here we create a link to point Apollo Client to our GraphQL API running at `localhost:4001/graphql`. Later, we'll use the link to add an authorization header that includes an auth token once a user has signed in to the application, but for now we'll start with making unauthenticated GraphQL requests.

## Making GraphQL Queries With Apollo Client In A Next.js Page

With Next.js' file based routing system we can create new pages just by creating a file in the `pages` directory and exporting a React component. Let's create a `podcasts` page to display a list of available podcasts. We'll make use of the `useQuery` Apollo Client hook to execute a GraphQL query to find all podcasts in the database and return their title.

```js
import { useQuery, gql } from '@apollo/client'

const PodcastQuery = gql`
  {
    Podcast {
      title
    }
  }
`

const Podcasts = () => {
  const { data } = useQuery(PodcastQuery)

  return (
    <div>
      <ul>
        {data?.Podcast.map((v) => {
          return <li key={v.title}>{v.title}</li>
        })}
      </ul>
    </div>
  )
}

export default Podcasts
```

Next.js will automatically add a route at `/podcasts` to render our new page.

<Image
  src="/static/images/grandcast-4/podcasts.png"
  alt="List of podcasts"
  height={698}
  width={1658}
/>

There are only a few podcasts in the database since they are only added to the database when a user subscribes. Refer to [Episode 2](https://lyonwj.com/blog/grandstack-podcast-app-user-auth-podcast-subscribe#podcast-subscribe) to see how we implemented podcast subscribe functionality.

## Slight Detour - Setting Up The GraphQL VS Code Extension

If you use the VS Code editor it can be helpful to make use of the GraphQL VS Code extension. This extension will enable syntax highlighting in .graphql files, gql template tags, and can also be used to execute GraphQL queries that are tagged in your JavaScript files. To use the GraphQL VS Code extension you'll need to install it from the extension marketplace [here.](https://marketplace.visualstudio.com/items?itemName=GraphQL.vscode-graphql) Then you'll also need to configure your project's GraphQL APIs in a GraphQL config file. Here's a simple example for this project:

```yml
schema: 'api/src/schema.graphql'
extensions:
  endpoints:
    default:
      url: http://localhost:4001/graphql
```

This will now allow us to execute GraphQL queries from VS Code that we've tagged with the `gql` template tag:

<Image
  src="/static/images/grandcast-4/vscode-extension.png"
  alt="GraphQL VS Code extension"
  height={1382}
  width={3422}
/>

# Authenticated GraphQL Requests In Next.js

We've seen how to make use of Apollo Client in our Next.js application to render a list of available podcasts, however this only works for unauthenticated GraphQL requests. We need to allow users to sign in to our application and then once they've signed in expose the user-specific functionality that we've built into our GraphQL API layer: podcast subscribe, new episode feeds, and create/view/update episode playlists.

To handle authentication and the associated state (Is the user signed in? What is the user's auth token?) we will make use of the React Context API and the Provider pattern again, this time creating our own `AuthProvider` that will allow us to keep track of and update authentication related state. Specifically, we'll need to:

* Execute the `login` GraphQL mutation operation to sign a user into the application.
* The `login` GraphQL mutation will return a JWT token when a user successfully signs in. We'll need to keep track of that token.
* When a user is signed in and a valid JWT is generated we need to add the token to all GraphQL requests as an authentication header.
* Allow a user to sign out.

All of this functionality will then be made available to any of our application components by importing the `useAuth` hook that we will create in our authentication provider.


```js
import React, { useState, useContext, createContext } from 'react'
import {
  ApolloProvider,
  ApolloClient,
  InMemoryCache,
  HttpLink,
  gql,
} from '@apollo/client'

const authContext = createContext()

export function AuthProvider({ children }) {
  const auth = useProvideAuth()

  return (
    <authContext.Provider value={auth}>
      <ApolloProvider client={auth.createApolloClient()}>
        {children}
      </ApolloProvider>
    </authContext.Provider>
  )
}

export const useAuth = () => {
  return useContext(authContext)
}

function useProvideAuth() {
  const [authToken, setAuthToken] = useState(null)

  const isSignedIn = () => {
    if (authToken) {
      return true
    } else {
      return false
    }
  }

  const getAuthHeaders = () => {
    if (!authToken) return null

    return {
      authorization: `Bearer ${authToken}`,
    }
  }

  const createApolloClient = () => {
    const link = new HttpLink({
      uri: 'http://localhost:4001/graphql',
      headers: getAuthHeaders(),
    })

    return new ApolloClient({
      link,
      cache: new InMemoryCache(),
    })
  }

  const signIn = async ({ username, password }) => {
    const client = createApolloClient()
    const LoginMutation = gql`
      mutation signin($username: String!, $password: String!) {
        login(username: $username, password: $password) {
          token
        }
      }
    `

    const result = await client.mutate({
      mutation: LoginMutation,
      variables: { username, password },
    })

    console.log(result)

    if (result?.data?.login?.token) {
      setAuthToken(result.data.login.token)
    }
  }

  const signOut = () => {
    setAuthToken(null)
  }

  return {
    setAuthToken,
    isSignedIn,
    signIn,
    signOut,
    createApolloClient,
  }
}

```

Note how we make use of the `HttpLink` to add the authentication header to GraphQL requests when a user is signed in. We also wrap the `ApolloProvider` component in our `AuthProvider` which will make Apollo Client and the authentication functionality available to any child components in the React component hierarchy. We'll need to update `_app.js` to use this new `AuthProvider`:

```js
import '../styles/globals.css'
import { AuthProvider } from '../lib/auth.js'

function MyApp({ Component, pageProps }) {
  return (
    <AuthProvider>
      <Component {...pageProps} />
    </AuthProvider>
  )
}

export default MyApp
```

## The Sign In Flow

Now that we've implemented authentication functionality in our `AuthProvider` we can implement the sign in flow in our index page. First, we'll want to import the new `useAuth` hook we created:


```js
import { useAuth } from '../lib/auth.js'
```

Now, let's add a `SignIn` component with a form for the user to submit their username and password. When the user submits the form we'll call the `signIn` function made available in `auth.js` which will execute the `login` GraphQL mutation and generate an auth JSON Web Token (JWT).

```js
const SignIn = () => {
  const [username, setUsername] = useState('')
  const [password, setPassword] = useState('')
  
  const { signIn, signOut } = useAuth()

  function onSubmit(e) {
    e.preventDefault()
    signIn({ username, password })
  }

  return (
    <div>
      <form onSubmit={onSubmit}>
        <input
          type="text"
          placeholder="username"
          onChange={(e) => setUsername(e.target.value)}
        ></input>
        <input
          type="password"
          placeholder="password"
          onChange={(e) => setPassword(e.target.value)}
        ></input>
        <button type="submit">Sign In</button>
      </form>
    </div>
  )
}
```

Once a user is signed in we want to show their podcast episode feed - the most recent podcast episodes across all the podcasts they subscribe to. This is available via the `episodeFeed` GraphQL query field. We'll create a `FeedQuery` component to execute this query and render the results in a list. Since the user is signed in, the appropriate auth token will be added in the header of the GraphQL request, making this an authenticated GraphQL request against our API.

```js
const FeedQuery = gql`
  {
    episodeFeed(first: 50) {
      id
      title
      audio
      podcast {
        title
      }
    }
  }
`

const EpisodeFeed = () => {
  const { data } = useQuery(FeedQuery)
  const { signOut } = useAuth()
  return (
    <div>
      <h1>Episode Feed</h1>
      <ul>
        {data?.episodeFeed.map((v) => {
          return <li key={v.id}>{v.title}</li>
        })}
      </ul>
      <button onClick={() => signOut()}>Sign Out</button>
    </div>
  )
}
```

Next, we'll make use of the `isSignedIn` function made available via `useAuth` to determine whether to render the sign in form or the episode feed component.

```js
export default function Home() {
  const { isSignedIn } = useAuth()
  return (
    <div className={styles.container}>
      <Head>
        <title>GRANDcast.FM</title>
        <link rel="icon" href="/favicon.ico" />
      </Head>

      <main className={styles.main}>
        <h1>GRANDcast.FM</h1>
        {!isSignedIn() && <SignIn />}
        {isSignedIn() && <EpisodeFeed />}
      </main>
    </div>
  )
}
```

Now, when we first load the page we're presented with a sign in form that will execute the `login` GraphQL mutation when submitted. If a successful login, then the GraphQL server will generate an authorization JWT and return the token to the client. Our auth provider will store that token and attach it in the authorization header of all future GraphQL requests, until we log out.

<Image
  src="/static/images/grandcast-4/signin-form.png"
  alt="List of podcast episodes"
  height={926}
  width={1638}
/>

Once we're authenticated, instead of the login form, we're presented with a list of the most recent podcast episodes for all podcasts to which we subscribe. We can click the "Sign Out" button, which will update the authentication state in our application, remove the token from future GraphQL requests, and we'll be presented with the sign in form again.

<Image
  src="/static/images/grandcast-4/podcast-list.png"
  alt="List of podcast episodes"
  height={1976}
  width={1892}
/>

Now that we've set up client-side authentication in our Next.js application we can start to explore the different data-fetching patterns available to us in Next.js. We skipped over this functionality in this post but the different server data-fetching models in Next.js are some of the most interesting features of the framework. We'll dig into this in the next episode. 

Be sure to subscribe to the Neo4j livestream on [Twitch](https://www.twitch.tv/neo4j) or [Youtube](https://www.youtube.com/neo4j) to keep up to date as we build out more features in [GRANDcast.FM](https://grandcast.fm). What else would you like us to explore on the livestream? Let us know on [Twitter.](https://twitter.com/lyonwj) And please drop any comments and feedback specific to this post over at [Dev.to](https://dev.to/lyonwj/getting-started-with-next-js-and-graphql-authentication-building-grandcast-fm-episode-4-14em)

# Resources

* [Code on Github](https://github.com/johnymontana/grandcast.fm)
* [Next.js Tutorial](https://nextjs.org/learn/basics/create-nextjs-app)
* [Neo4j Next.js Example](https://github.com/vercel/next.js/tree/canary/examples/with-neo4j)
* [Neo4j Next.js Example With GraphQL](https://github.com/vercel/next.js/tree/canary/examples/with-apollo-neo4j-graphql)
* [GraphQL VSCode Extension](https://marketplace.visualstudio.com/items?itemName=GraphQL.vscode-graphql)

Building A GRANDstack Podcast App: Episode 4

Getting Started With Next.js and GraphQL Authentication


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

> This is the third post in a series about building a podcast application using [GRANDstack](https://grandstack.io). Check out the first post, ["Podcast Search GraphQL API With Neo4j And The Podcast Index"](https://lyonwj.com/blog/grandstack-podcast-app-podcast-search-graphql-api) where we start building the GraphQL API and implement podcast search functionality and the second post, ["GRANDcast.FM: User Auth & Podcast Subscribe Functionality"](https://lyonwj.com/blog/grandstack-podcast-app-user-auth-podcast-subscribe)

<Subscribe />

In this post we continue building out functionality in the GraphQL API powering our podcast application. This week we focus on adding episodes by parsing podcast XML feeds and adding user playlists to the GraphQL API. We added this functionality during the Neo4j livestream. You can watch the recording of that here:

<YouTube youTubeId="WdClRB_-Egs" />

# Working With Podcast Feeds

A podcast feed is an XML document that conforms to the [RSS 2.0 specification.](https://validator.w3.org/feed/docs/rss2.html) The basic structure of an RSS document consists of a top-level `<rss>` element, followed by a `<channel>` element which contains metadata about the podcast (title, description, link, etc). We already have this data for each podcast that a user has subscribed to, fetched from the Podcast Index API and added to the database. What we're interested in is the next sub-element of the document, `<items>`, which consists of one or more `<item>` elements.

Each `<item>` element represents a podcast episode and contains the following elements (technically these are all optional according to the specification, and other elements may be included as well):

- `<title>` - The title of the podcast episode
- `<link>` - The URL of the podcast episode (not to be confused with the audio file URL)
- `<description>` - A synopsis of the episode
- `<enclosure>` - A media object (such as an audio file) attached to the episode
- `<guid>` - A globally unique string used to identify the episode
- `<pubDate>` - Indicates when the episode was published, using RFC822 datetime format.

Now that we know the structure of a podcast feed XML document, let's start parsing them and adding episodes to the database.

## Parsing XML With Neo4j

We can parse and import XML data directly into Neo4j using the [`apoc.load.xml` procedure.](https://neo4j.com/labs/apoc/4.1/import/xml/) This procedure takes the URL for the XML file and returns a single nested map representing the XML document using `_type`, `_text`, and `_children` keys to represent values, metadata, and element type as represented in XML. The [APOC documentation](https://neo4j.com/labs/apoc/4.1/import/xml/#load-xml-examples-local-file) has a simple example showing how this works. Here we use Cypher to filter for the elements of each episode that we want to capture, using the [Graphistania podcast](https://podcasts.apple.com/us/podcast/graphistania-podcast-for-neo4j-graph-database-community/id975377379) as an example:

```cypher
MATCH (p:Podcast)
CALL apoc.load.xml("http://feeds.soundcloud.com/users/soundcloud:users:141739624/sounds.rss")
YIELD value
WITH [x in value._children[0]._children WHERE x._type = "item"] AS episodes
UNWIND episodes AS episode
WITH
	[X in episode._children WHERE X._type = "title"][0]._text AS title,
	[X in episode._children WHERE X._type = "summary"][0]._text AS summary,
	[X in episode._children WHERE X._type = "link"][0]._text AS link,
	[X in episode._children WHERE X._type = "image"][0].href AS image,
	[X in episode._children WHERE X._type = "enclosure" AND X.type CONTAINS "audio"][0].url AS audio,
	[X in episode._children WHERE X._type = "pubDate"][0]._text AS pubDate,
	[X in episode._children WHERE X._type ="guid"][0]._text AS guid
RETURN *
```

| audio | guid | image | link | pubDate | summary | title |
|-------|------|-------|------|---------|---------|-------|
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/935068060-graphistania-graphistania-20-episode-11.mp3" | "tag:soundcloud,2010:tracks/935068060" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-11" | "Tue, 24 Nov 2020 00:00:00 +0000" | "Graphistania 2.0 - Episode 11 by The Neo4j Graph Database Community" | "Graphistania 2.0 - Episode 11" |
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/926771035-graphistania-graphistania-20-episode-10.mp3" | "tag:soundcloud,2010:tracks/926771035" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-10" | "Thu, 12 Nov 2020 08:08:48 +0000" | "Graphistania 2.0 - Episode 10 by The Neo4j Graph Database Community" | "Graphistania 2.0 - Episode 10" |
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/905615800-graphistania-graphistania-20-episode-9.mp3" | "tag:soundcloud,2010:tracks/905615800" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-9" | "Tue, 06 Oct 2020 00:00:00 +0000" | "Graphistania 2.0 - Episode 9 by The Neo4j Graph Database Community" | "Graphistania 2.0 - Episode 9" |
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/889154860-graphistania-graphistania-20-episode-8.mp3" | "tag:soundcloud,2010:tracks/889154860" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-8" | "Tue, 08 Sep 2020 00:00:00 +0000" | "Graphistania 2.0 - Episode 8 by The Neo4j Graph Database Community" | "Graphistania 2.0 - Episode 8" |
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/853044658-graphistania-graphistania-20-episode-7.mp3" | "tag:soundcloud,2010:tracks/853044658" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-7" | "Fri, 10 Jul 2020 00:00:00 +0000" | "Graphistania 2.0 - episode 7 by The Neo4j Graph Database Community" | "Graphistania 2.0 - episode 7" |
| "http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/791209915-graphistania-graphistania-20-episode-6.mp3" | "tag:soundcloud,2010:tracks/791209915" | "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png" | "https://soundcloud.com/graphistania/graphistania-20-episode-6" | "Mon, 06 Apr 2020 00:00:00 +0000" | "Graphistania 2.0 - episode 6 by The Neo4j Graph Database Community" | "Graphistania 2.0 - episode 6" |

Now that we can parse each podcast feed we're ready to add these episodes to the database.

## Adding Podcast Episodes To The Graph

We'll use the `apoc.load.xml` procedure to parse the podcast feeds for any podcasts currently stored in the database and create `Episode` nodes in the graph with the relevant details for each episode stored as node properties. We'll connect the episode to its podcast using a `HAS_EPISODE` relationship.

<Image
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/datamodel.png"
  alt="Graph data model for podcast episodes and playlists"
  width={1191}
  height={897}
/>


Here we create an `Episode` node for each podcast episode parsed from the feed. Note how we make use of the `apoc.date.parse` function when storing the `pubDate` property as a datetime type.

```cypher
MATCH (p:Podcast)
call apoc.load.xml(p.feedURL)
YIELD value
UNWIND value._children AS foo
WITH p, [x in foo._children WHERE x._type = "item"] AS episodes
UNWIND episodes AS episode
WITH p,
       [X in episode._children WHERE X._type = "title"][0]._text AS title,
       [X in episode._children WHERE X._type = "summary"][0]._text AS summary,
       [X in episode._children WHERE X._type = "link"][0]._text AS link,
       [X in episode._children WHERE X._type = "image"][0].href AS image,
       [X in episode._children WHERE X._type = "enclosure" AND X.type CONTAINS "audio"][0].url AS audio,
       [X in episode._children WHERE X._type = "pubDate"][0]._text AS pubDate,
       [X in episode._children WHERE X._type ="guid"][0]._text AS guid

MERGE (e:Episode {id: guid})
SET e.title = title,
    e.summary = summary,
    e.link = link,
    e.image = image,
    e.audio = audio,
    e.pubDate = datetime({epochMillis: apoc.date.parse(pubDate, "ms", "EEE, dd MMM yyyy HH:mm:ss z")})
MERGE (e)<-[:HAS_EPISODE]-(p)
```

## Adding Episodes When A User Subscribes To A Podcast

Now that we have the Cypher statement to parse podcast feeds and insert episodes into the database we need to update our GraphQL API so that when a podcast is added to the database we start parsing the feed. There are a few different ways we could approach this, perhaps creating a [database trigger](https://neo4j.com/labs/apoc/4.1/background-operations/triggers/) or a [background job](https://neo4j.com/labs/apoc/4.1/background-operations/periodic-background/) to parse the feed. Instead of those options let's update the `subscribeToPodcast` GraphQL mutation to include the Cypher for parsing the podcast feed and adding episodes to the graph when a user subscribes to the podcast. To do that we'll add the above Cypher statement to the `subscribeToPodcast` mutation `@cypher` statement:

```graphql
type Mutation {
  signup(username: String!, password: String!): AuthToken
  login(username: String!, password: String!): AuthToken
  subscribeToPodcast(itunesId: String!): Podcast
  @cypher(
    statement: """
    // First we fetch the podcast details from the podcast index
    WITH toString(timestamp()/1000) AS timestamp
    WITH {
    `User-Agent`: 'GRANDstackFM',
    `X-Auth-Date`: timestamp,
    `X-Auth-Key`: apoc.static.get('podcastkey'),
    `Authorization`: apoc.util.sha1([apoc.static.get('podcastkey')+apoc.static.get('podcastsecret') +timestamp])
    } AS headers
    CALL apoc.load.jsonParams('https://api.podcastindex.org/api/1.0/podcasts/byitunesid?id=' + apoc.text.urlencode($itunesId), headers, '', '') YIELD value
    WITH value.feed AS feed

    // Next we create the (:User)-[:SUBSCRIBES_TO]->(:Podcast) path in the database
    MATCH (u:User {id: $cypherParams.userId})
    MERGE (p:Podcast {itunesId: $itunesId})
    SET p.title       = feed.title,
        p.link        = feed.link,
        p.description = feed.description,
        p.feedURL     = feed.url,
        p.image       = feed.artwork
    MERGE (u)-[:SUBSCRIBES_TO]->(p)
    WITH p

    // Now we parse the podcast XML feed and add its episodes to the database
    // (:Podcast)-[:HAS_EPISODE]->(:Episode)
    CALL apoc.load.xml(p.feedURL) YIELD value
    UNWIND value._children AS foo
    WITH p,[x in foo._children WHERE x._type = \"item\"] AS episodes
    UNWIND episodes AS episode
    WITH p,[x in episode._children WHERE x._type =\"title\"][0]._text AS title,
        [x in episode._children WHERE x._type =\"description\"][0]._text AS summary,
        [x in episode._children WHERE x._type=\"link\"][0]._text AS link,
        [x in episode._children WHERE x._type=\"image\"][0].href AS image,
        [x in episode._children WHERE x._type=\"enclosure\" AND x.type CONTAINS \"audio\" ][0].url AS audio,
        [x in episode._children WHERE x._type=\"pubDate\"][0]._text AS pubDate,
        [x in episode._children WHERE x._type =\"guid\"][0]._text AS guid
    MERGE (e:Episode {id: guid})
    SET e.title = title,
        e.summary = summary,
        e.link = link,
        e.image = image,
        e.audio = audio,
        e.pubDate = dateTime({epochMillis: apoc.date.parse(pubDate, 'ms', 'EEE, dd MMM yyyy HH:mm:ss zzz')})
    MERGE (e)<-[:HAS_EPISODE]-(p)
    RETURN p
    """
  )
}
```

We'll also want to add the `Episode` type definition to our GraphQL API:

```graphql
type Episode {
  id: ID!
  pubDate: DateTime
  summary: String
  title: String
  link: String
  image: String
  audio: String
  podcast: Podcast @relation(name: "HAS_EPISODE", direction: "IN")
}
```

## User Podcast Feed

A typical flow for our application will be:

1. Show the user a feed of episodes for all the podcasts to which they subscribe, most recent first
1. The user adds episodes they want to listen to to a playlist
1. When the user is ready to listen they can view all episodes assigned to each playlist

Let's add a GraphQL query field to show all episodes for each podcast a user subscribes to, showing the most recent episodes first. We'll use the `cypherParams.userId` value, which will identify the currently authenticated user, to look up that user in the database, traverse to all podcasts they subscribe to, and then traverse to each episode for their subscribed podcasts. Refer to [the previous post in this series](https://lyonwj.com/blog/grandstack-podcast-app-user-auth-podcast-subscribe) for details about the `cypherParams.userId` value and how that is parsed from the auth token.

```graphql
type Query {
  episodeFeed(first: Int = 10, offset: Int = 0): [Episode]
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})-[:SUBSCRIBES_TO]->(p:Podcast)-[:HAS_EPISODE]->(e:Episode)
    RETURN e ORDER BY e.pubDate DESC SKIP toInteger($offset) LIMIT toInteger($first)
    """
  )
}
```

First, we'll log in using the username and password we created previously:

<Image 
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/graphql1.png"
  alt="Logging in using GraphQL"
  width={3476}
  height={906}
/>

This gives us the auth token that when added to the request header will allow us to make authenticated GraphQL requests to the API. Now, let's make an authenticated request to query for our user's podcast episode feed, showing the most recent episodes across all the podcasts I subscribe to:

<Image 
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/graphql2.png"
  alt="User podcast episode feed"
  width={2996}
  height={1694}
/>


# Playlists

Next, let's introduce the concept of playlists to our API. We want to add individual episodes to one or more playlists, which we should then be able to query for later and play episodes from each playlist. First, we'll add the `Playlist` type to our GraphQL type definitions:

```graphql
type Playlist {
  name: String!
  episodes: [Episode] @relation(name: "IN_PLAYLIST", direction: "IN")
}
```

We want playlists to be private to the currently authenticated user so we'll add the `Playlist` type to the configuration to be excluded from automatic query and mutations. Instead we'll create `@cypher` directive fields to access playlists.

```js
const schema = makeAugmentedSchema({
  typeDefs,
  resolvers,
  config: {
    query: {
      exclude: ['PodcastSearchResult', 'AuthToken', 'User', 'Playlist']
    },
    mutation: {
      exclude: ['PodcastSearchResult', 'AuthToken', 'User', 'Playlist']
    }
  }
});
```

## Create Playlist

Since we added the `Playlist` type to be excluded from the generated queries and mutations during the schema augmentation process we'll need to define the logic for creating and updating playlists using `@cypher` schema directive fields. Let's add a `createPlaylist` field to the `Mutation` type:

```graphql
type Mutation {
  createPlaylist(name: String!): Playlist
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})
    MERGE (p:Playlist {name: $name})<-[:OWNS]-(u)
    RETURN p
    """
  )
}
```

Now, we can create a playlist. Let's create a playlist called "jogging" that I'll use to listen to episodes while I'm on jogs around the neighborhood:

<Image 
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/graphql3.png"
  alt="Creating a playlist"
  width={2762}
  height={1630}
/>

Note that the episodes array is empty - we haven't added any episodes to this playlist yet. Let's add that functionality to the GraphQL API now.

## Add Episodes To Playlist

To enable users to add episodes to each playlist we'll create a new mutation field called `addEpisodeToPlaylist` that will take as arguments the name of the playlist and the id of a podcast episode to be added to the playlist. We'll define the logic for this mutation using a `@cypher` schema directive, ensuring that we match on the authenticated user and make sure they are the owner of the playlist before adding an `IN_PLAYLIST` relationship connecting the `Episode` and `Playlist` nodes.

```graphql
type Mutation {
  addEpisodeToPlaylist(name: String!, podcastId: ID!): Playlist
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})-[:OWNS]->(p:Playlist {name: $name})
    MATCH (e:Episode {id: $podcastId})
    MERGE (e)-[:IN_PLAYLIST]->(p)
    RETURN p
    """
  )
}
```

<Image 
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/graphql4.png"
  alt="Adding episodes to our playlist"
  width={2960}
  height={1494}
/>

## Get Playlist Episodes

Finally, we'll implement a query field to return all playlists a user owns, which can then also be used to return the episodes in each playlist, including episode details.

```graphql
type Query {
  playlists: [Playlist]
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})-[:OWNS]->(p:Playlist)
    RETURN p
    """
  )
}
```

<Image 
  src="/static/images/grandstack-podcast-app-parsing-xml-neo4j-rss-episodes-playlists/graphql5.png"
  alt="Adding episodes to our playlist"
  width={3220}
  height={1506}
/>

Thanks for reading! Please send any feedback or comments to me [on Twitter](https://twitter.com/lyonwj) or [over at Dev.to](https://dev.to/lyonwj/parsing-and-importing-xml-with-neo4j-adding-episodes-and-playlists-to-grandcast-fm-podcast-app-105k).

# Resources

* [Code on Github](https://github.com/johnymontana/grandcast.fm)
* [GRANDstack docs: Cypher parameters from context](https://grandstack.io/docs/neo4j-graphql-js-middleware-authorization#cypher-parameters-from-context)
* [APOC User Guide](https://neo4j.com/labs/apoc/4.2/)
* [Cypher Refcard](https://neo4j.com/docs/cypher-refcard/current/)
* [RSS 2.0 Specification](https://validator.w3.org/feed/docs/rss2.html)

Building A GRANDstack Podcast App: Episode 3

Parsing And Importing XML With Neo4j: Adding Episodes and Playlists To GRANDcast.FM Podcast App


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/grandcast.png"
  alt="GRANDcast.FM logo"
  width={1920}
  height={540}
/>

> This is the second post in a series about building a podcast application using [GRANDstack](https://grandstack.io). Check out the first post, ["Podcast Search GraphQL API With Neo4j And The Podcast Index"](/blog/grandstack-podcast-app-podcast-search-graphql-api) where we start building the GraphQL API and implement podcast search functionality.

<Subscribe />

In the [previous post](/blog/grandstack-podcast-app-podcast-search-graphql-api) we started our GRANDstack podcast application by creating the GraphQL API and adding podcast search functionality using the Podcast Index API. After searching for podcasts the next thing our users will want to do is start subscribing to them, so in this episode we focus on allowing users to sign up and log in to our application and then implement "subscribe to podcast" functionality. We built this functionality on the [Neo4j live stream](https://twitch.tv/neo4j_) which you can watch here:

<YouTube youTubeId="IOXmrIYij_g" />

# User Authentication

Let's start with implementing user authentication. We'll need to allow users to sign up for our application, create a username and choose a password. We'll also need to enable users to log in to the application and generate an authorization token for them.

## Setup

We'll make use of two packages to enable user authentication:

```shell
npm install jsonwebtoken bcrypt
```

* **[`bcrypt`](https://www.npmjs.com/package/bcrypt)** - an implementation of the bcrypt algorithm, a one-way hashing algorithm commonly used to hash passwords. We'll use this library to create a hash of a user's password when they register. We'll store the hash in the database and compare that to the password they submit when attempting to log in.

* **[`jsonwebtoken`](https://www.npmjs.com/package/jsonwebtoken)** a JavaScript implementation of JSON Web Token (JWT), a standard for cryptographically encoding JSON data into a token, which can then be used as an authorization token. We'll generate a signed JWT after a user successfully signs up or logs in which the user will then be able to use to make authenticated requests to our GraphQL API.

We'll also need to generate a random 256 bit secret to use for signing our tokens. By default `jsonwebtoken` will use the HS256 algorithm, we could also choose to use the RSA256 algorithm which uses public/private key pairs. We'll stick with the default HS256 algorithm and store the key as an environment variable by adding it to our `.env` file:

```shell
JWT_SECRET=<RANDOM_256_BIT_SECRET_HERE>
```

## Sign Up

First, we'll add a new `signup` mutation to our GraphQL type definitions. This mutation field will take two arguments: `username` and `password`. Our new `signup` mutation will return an `AuthToken` object with a single string field called `token`.

```graphql
type Mutation {
  signup(username: String!, password: String!): AuthToken
}

type AuthToken {
  token: String!
}
```

Next, we implement the resolver function for the `signup` mutation field. We haven't created any resolvers yet because we've been taking advantage of the resolvers generated for us by `neo4j-graphql.js` but because we want to execute some custom logic in JavaScript that we can't express in Cypher we'll need to implement this resolver function.

Our `signup` resolver will take the user's password and hash it using `bcrypt`, store that hashed password and username in the database along with a randomly generated user id, then create a signed JWT that will include the username and id in the token's payload. The client application will then be able to use this auth token to make authenticated requests against our GraphQL API.

```js
import jwt from 'jsonwebtoken';
import { compareSync, hashSync } from 'bcrypt';

const resolvers = {
  Mutation: {
    signup: (obj, args, context, info) => {
      args.password = hashSync(args.password, 10);
      const session = context.driver.session();

      return session
        .run(
          `CREATE (u:User) SET u += $args, u.id = randomUUID()
           RETURN u`,
          { args }
        )
        .then((res) => {
          session.close();
          const { id, username } = res.records[0].get('u').properties;

          return {
            token: jwt.sign({ id, username }, process.env.JWT_SECRET, {
              expiredIn: '30d'
            })
          };
        });
    }
  }
};
```

Now, if we execute the `signup` GraphQL mutation we can create new users and generate an authorization token for each user.

```graphql
mutation {
  signup(username: "jennycat", password: "feedme") {
    token
  }
}
```

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/graphql1.png"
  alt="GraphQL mutation for user sign up"
  width={3056}
  height={1020}
/>

We can also take the token, paste it into the JWT debugger at [jwt.io](https://jwt.io) to decode the payload to see what values are encoded in the token. We should see the username and the random id generated for the user.

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/jwt1.png"
  alt="Using the JWT debugger to inspect the payload of our token"
  width={3258}
  height={1334}
/>

And if we check in Neo4j we'll see the `User` node created in the database with the username, generated user id, and hashed password all stored in the database.

<div style={{ maxWidth: '75%', margin: 'auto' }}>
  <Image
    src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/db1.png"
    alt="The user node is created in the database"
    width={1382}
    height={1038}
  />
</div>

New users are now able to sign up and register an account, but we also need to implement user login functionality so returning users can authenticate.

## Login

Implementing user login will be similar to how we implemented user sign up, but instead of creating the user node in the database we want to look it up by username in the database and compare the provided password with the hashed password stored in the database to make sure the user is providing the correct password in the GraphQL mutation argument.

First, we add a `login` field to the Mutation type in our GraphQL type definitions:

```graphql
type Mutation {
  signup(username: String!, password: String!): AuthToken
  login(username: String!, password: String!): AuthToken
}
```

And implement the resolver function for this mutation field in `index.js`. We'll look up the user by username and compare the provided password with the hashed password stored in the database. If the password doesn't match we throw an error, otherwise we generate a JWT authorization token.

```js
const resolvers = {
  signup: (obj, args, context, info) => {...},
  login: (obj, args, context, info) => {
    const session = context.driver.session();

    return session
      .run(
        `MATCH (u:User {username: $username}))
        RETURN u LIMIT 1`,
        { username: args.username }
      )
      .then((res) => {
        session.close();

        const { id, username, password } = res.records[0].get('u').properties;
        if (!compareSync(args.password, password)) {
          throw new Error('Authorization Error');
        }

        return {
          token: jwt.sign({ id, username }, process.env.JWT_SECRET, {
            expiresIn: '30d'
          })
        };
      });
  }
};
```

Now let's test the login operation using the username and password we created in the user sign up mutation previously. If all is working we should see a new authorization token.

```graphql
mutation {
  login(username: "jennycat", password: "feedme") {
    token
  }
}
```

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/graphql2.png"
  alt="Using the JWT debugger to inspect the payload of our token"
  width={3102}
  height={996}
/>

# Authenticated GraphQL Requests

Now that we're able to register new users and allow users to sign in, we can start to handle authenticated requests and take into account the identity of the user in our application. To make authenticated GraphQL requests the auth token will be added to the request headers as an authorization bearer token.

```json
{
  "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6ImI3YjBmYmZkLTJjNzMtNDBiZS1hZGUxLTM1MjMzZWJhZDE5ZSIsInVzZXJuYW1lIjoiamVubnljYXQiLCJpYXQiOjE2MDc3NDI2MDUsImV4cCI6MTYxMDMzNDYwNX0._2xHiqrrwZR3ZXH9np9O2oWcx6iBsWd4OZLnd6DjtqY"
}
```

## Find The Authenticated User

First, let's add a GraphQL query field to return the currently authenticated user. To do this we'll need to figure out which user is making the authenticated GraphQL request.

### Cypher Params

We've already seen the powerful [`@cypher` directive functionality](https://grandstack.io/docs/graphql-custom-logic/#the-cypher-graphql-schema-directive) of the neo4j-graphql.js library that allows us to define custom logic using Cypher in our GraphQL schema. We saw that any arguments of the GraphQL field are passed into the Cypher statement as Cypher parameters. Now, we will take advantage of the ["Cypher Parameters" feature of the `@cypher` directive](https://grandstack.io/docs/neo4j-graphql-js-middleware-authorization#cypher-parameters-from-context) that allow us to pass values into the Cypher statement using the GraphQL context object.

**Any values in the `context.cypherParams` object will be available in Cypher queries using the `@cypher` schema directive.** This can be used to inject user specific information into these Cypher queries. 

Let's update our instantiation of `ApolloServer` where we specify the value of the context object. Instead of an object, we can also define the context object using a function. This function is called on each request and is passed the request object, which will include the authorization header when making an authenticated GraphQL request. We'll grab the authorization token from the request header, validate it, and add the user id into the `cypherParams` object (remember that each token encodes the user id and username). This user id will then be available within the Cypher query when using the `@cypher` schema directive.

```js
const server = new ApolloServer({
  context: ({ req }) => {
    const token = req?.headers?.authorization?.slice(7);
    let userId;

    if (token) {
      const decoded = jwt.verify(token, process.env.JWT_SECRET);
      userId = decoded.id;
    }
    return {
      cypherParams: { userId },
      driver,
      neo4jDatabase: process.env.NEO4J_DATABASE
    };
  },
  schema
});
```

Now, if an authorization token is specified in the GraphQL request, we can reference `$cypheParams.userId` in our Cypher query to refer to the currently authenticated user. 

Let's add a GraphQL query field to return the current user, looking them up in the database by user id.

```graphql
type Query {
  currentUser: User
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})
    RETURN u
    """
  )
}

type User {
  username: String
  id: ID!
}
```

We can now query for the currently authenticated user.

```graphql
{
  currentUser {
    username
    id
  }
}
```

## Podcast Subscribe

Now that we're able to make authenticated requests to the API we can implement functionality that allows a user to subscribe to podcasts. A user should only be able to see their own podcast subscriptions and should not be able to subscribe podcasts for other users.

### Subscribe Mutation

We'll add one more GraphQL mutation field, `subscribeToPodcast`, which will take a single argument, the `itunesId` of a podcast, and subscribe the user to the podcast. However, we haven't yet stored any podcast data in the database - our podcast search functionality is calling the Podcast Index API and returning results but not updating the database. In the `subscribeToPodcast` mutation we want to make sure we have the podcast details to store in the database so we'll first make a call to the Podcast Index to fetch the podcast details, store the details in a `Podcast` node, then create a `SUBSCRIBES_TO` relationship connecting the `User` node and `Podcast` node. We also add a `Podcast` type to our GraphQL type definitions to represent these nodes.

```graphql
type Mutation {
  signup(username: String!, password: String!): AuthToken
  login(username: String!, password: String!): AuthToken
  subscribeToPodcast(itunesId: String!): Podcast
  @cypher(
    statement: """
    WITH toString(timestamp()/1000) AS timestamp
    WITH {
    `User-Agent`: 'GRANDstackFM',
    `X-Auth-Date`: timestamp,
    `X-Auth-Key`: apoc.static.get('podcastkey'),
    `Authorization`: apoc.util.sha1([apoc.static.get('podcastkey') + apoc.static.get('podcastsecret') + timestamp])
    } AS headers
    CALL apoc.load.jsonParams('https://api.podcastindex.org/api/1.0/podcasts/byitunesid?id=' + apoc.text.urlencode($itunesId), headers, '', '') YIELD value
    WITH value.feed AS feed
    MATCH (u:User {id: $cypherParams.userId})
    MERGE (p:Podcast {itunesId: $itunesId})
    SET p.title       = feed.title,
        p.link        = feed.link,
        p.description = feed.description,
        p.feedURL     = feed.url,
        p.image       = feed.artwork
    MERGE (u)-[:SUBSCRIBES_TO]->(p)
    RETURN p
    """
  )
}

type Podcast {
  itunesId: ID!
  title: String
  link: String
  feedURL: String
  description: String
  image: String
}
```

After a few users log in and start subscribing to podcasts our graph starts to look like this:

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/db2.png"
  alt="The graph of users and podcasts"
  width={2288}
  height={1034}
/>

### Get List of Subscribed Podcasts For The Authenticated User

We also want to return the list of podcasts the authenticated user has subscribed to. To do this we add a `subscribedPodcasts` query field that will use the `$cypherParams.userID` value to find the `User` node in the database and any subscribed podcasts for the user.

```graphql
type Query {
  subscribedPodcasts: [Podcast]
  @cypher(
    statement: """
    MATCH (u:User {id: $cypherParams.userId})-[:SUBSCRIBES_TO]->(p:Podcast)
    RETURN p
    """
  )
}
```

We can now query for our subscribed podcasts - be sure to include our authorization token as an authorization header.

```graphql
{
  subscribedPodcasts {
    title
    description
    itunesId
    feedURL
    image
  }
}
```

<Image
  src="/static/images/grandstack-podcast-app-user-auth-podcast-subscribe/graphql3.png"
  alt="Subscribed podcasts"
  width={3212}
  height={1646}
/>

We've now implemented user registration, log in, and podcast subscriptions in our GraphQL API. There are a few more cases we'll need to address (making sure users can't register the same username, better error handling, handling password reset, etc) but we now have basic authentication in our GraphQL API. In the next episode we'll start parsing feed URLs and adding episodes and playlists to the graph.

# Resources

- [Code on Github](https://github.com/johnymontana/grandcast.fm)
- [GRANDstack docs: Cypher parameters from context](https://grandstack.io/docs/neo4j-graphql-js-middleware-authorization#cypher-parameters-from-context)
- [`brcypt` npm package](https://www.npmjs.com/package/bcrypt)
- [`jsonwebtoken` npm package](https://www.npmjs.com/package/jsonwebtoken)
- [JWT debugger](https://jwt.io/)
- [Random key generator](https://randomkeygen.com/)


Building A GRANDstack Podcast App: Episode 2

GRANDcast.FM: User Auth & Podcast Subscribe Functionality 


import Image from 'next/image';
import Subscribe from '../../components/Subscribe';
import YouTube from '../../components/YouTube';

<div style={{ maxWidth: '50%', margin: 'auto' }}>
  <Image
    src="/static/images/grandstack-podcast-app-podcast-search-graphql-api/grandcast.png"
    alt="GRANDcast.FM logo"
    width={866}
    height={534}
  />
</div>

A few weeks ago I was complaining about the podcast app I was using. There was some problem with updating my playlists and I was grumpy because the app wasn't pulling down any episodes. My wife suggested I just build my own podcast application and use that instead. She was just trying to get me to shut up and stop complaining, but I thought that actually sounded like a fun project. So this week on the [Neo4j livestream](https://twitch.tv/neo4j_) we kicked off a new project: building a podcast application with [GRANDstack!](https://grandstack.io) 

<Subscribe />

In this first week we went over the graph data model we'll need for this application and started building the GraphQL API for the project, focusing on implementing podcast search functionality using the Podcast Index API. This blog post covers what we built in that stream, but you can watch the recording of the stream here:

<YouTube youTubeId="lsGgFxGwHlw" />

# Graph Data Modeling

When beginning a new project I always like to start with a data modeling exercise. In my mind graph data modeling is an iterative process that begins as soon as you start defining the business requirements of the application. The steps of the graph data modeling iterative process:

1. Identify the entities. These become **nodes.**
2. How are these entities connected? These connections become **relationships.**
3. What attributes describe the entities? These become **node properties.**
4. What attributes are specific to how entities are connected? These become **relationship properties.**

Once you have an initial model, work through the business requirements and try to identify a traversal through the graph model that fulfills the requirement. If you can't traverse the graph to answer the necessary question, you may need to update the graph model.

After going through this process initially we ended up with a graph model that looks like this (created using the [Arrows graph diagramming tool.](http://www.apcjones.com/arrows/))

<Image
  src="/static/images/grandstack-podcast-app-podcast-search-graphql-api/datamodel.png"
  alt="Building a podcast application livestream"
  width={2568}
  height={1648}
/>

# Podcast Search

The first feature I wanted to implement is podcast search since that's the first thing our users will want to do. Our [GRANDstack application](https://grandstack.io) will be powered by a GraphQL API so we want to expose this podcast search functionality in our GraphQL API. We'll make use of GRANDstack's [`@cypher` schema directive](https://grandstack.io/docs/graphql-custom-logic/#the-cypher-graphql-schema-directive) functionality for adding custom logic to implement this podcast search functionality, but instead of searching a Neo4j database, we'll query a 3rd party API endpoint to search for podcasts.

## Podcast Index

<div style={{maxWidth: "65%", margin: "auto"}}>
<Image 
  src="/static/images/grandstack-podcast-app-podcast-search-graphql-api/podcastindex.jpg"
  alt="The Podcast Index"
  width={2048}
  height={771}
/>
</div>

We don't don't want to maintain an index of all possible podcasts in existence, instead we'll use a 3rd party API to enable searching for podcasts, in this case a service call [the Podcast Index.](https://podcastindex.org/) The Podcast Index is committed to maintaining an open API of podcasts for the preservation of free speech. In line with its goal anyone can register for an API token and begin building application using its API.

### Searching Endpoint

Looking at the [API docs for the Podcast Index](https://podcastindex-org.github.io/docs-api/) we can see there is an API endpoint for searching for podcasts. To use this endpoint we include a single query parameter which is our search term:

```json
{
  "status": "true",
  "feeds": [
    {
      "id": 969306,
      "title": "Graphistania: Podcast for Neo4j Graph Database community",
      "url": "http://feeds.soundcloud.com/users/soundcloud:users:141739624/sounds.rss",
      "originalUrl": "http://feeds.soundcloud.com/users/soundcloud:users:141739624/sounds.rss",
      "link": "http://blog.bruggen.com",
      "description": "Podcast by The Neo4j Graph Database Community",
      "author": "The Neo4j Graph Database Community",
      "ownerName": "Graphistania",
      "image": "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png",
      "artwork": "http://i1.sndcdn.com/avatars-000135096101-qekfg1-original.png",
      "lastUpdateTime": 1606220307,
      "lastCrawlTime": 1607132366,
      "lastParseTime": 1606220308,
      "lastGoodHttpStatusTime": 1607132366,
      "lastHttpStatus": 200,
      "contentType": "application/rss+xml; charset=utf-8",
      "itunesId": 975377379,
      "generator": null,
      "language": "en",
      "type": 0,
      "dead": 0,
      "crawlErrors": 0,
      "parseErrors": 0,
      "categories": {
        "102": "Technology"
      },
      "locked": 0,
      "imageUrlHash": 775621088
    }
  ],
  "count": 1,
  "query": "neo4j",
  "description": "Found matching feeds."
}
```
### Authorization Headers

According to the [Podcast Index API documentation](https://podcastindex-org.github.io/docs-api/) each request need to be authenticated using "Amazon style" request authorization token headers. This means that each request must include the following headers:

* `User-Agent` - to identify our application "GRANDcast.FM"
* `X-Auth-Date` - the current unix epoch time, expressed in seconds.
* `X-Auth-Key` - our API key. We can get a free API key [simply by registering.](https://api.podcastindex.org/)
* `Authorization` - a SHA1 hash of our API key, API secret, and the current unix epoch time concatenated together

Since we want to use the `@cypher` schema directive to query the Podcast Index, we'll need to format and parse the request to this API using Cypher. How can we make a HTTP request, format the request headers, and parse a JSON response with Cypher?

## APOC To The Rescue

Fortunately the [APOC standard library for Neo4j](https://neo4j.com/labs/apoc/) has some helpful procedures and functions that will enable us to query the Podcast Index API with Cypher. We'll make use of the follow APOC procedures and functions:

- [**`apoc.load.jsonParams`**](https://neo4j.com/labs/apoc/4.1/overview/apoc.load/apoc.load.jsonParams/) - load data from a JSON URL, including passing a request payload and our authorization headers
- [**`apoc.static.get`**](https://neo4j.com/labs/apoc/4.0/misc/static-values/) - reading static values such as API credentials from a config file so we don't have to check our secrets into version control
- [**`apoc.util.sha1`**](https://neo4j.com/labs/apoc/4.1/overview/apoc.util/apoc.util.sha1/) - compute the SHA1 hash of a string. We'll use this to compute the hash for the `Authorization` token in our request.
- [**`apoc.text.urlencode`**](https://neo4j.com/labs/apoc/4.0/overview/apoc.text/apoc.text.urlencode/) - encode text so it can be safely used in a URL. We use this URL encode the search term parameter.
- [**`apoc.map.values`**](https://neo4j.com/labs/apoc/4.1/overview/apoc.map/apoc.map.values/) - convert a map into a list of values. We'll use this to work with the categories data for podcast search results.

First, we'll add our API credentials from Podcast Index to `apoc.conf`:

```shell
apoc.static.podcastkey=<YOUR_API_KEY_HERE>
apoc.static.podcastsecret=<YOUR_API_SECRET_HERE>

```

We'll now be able to reference these values using the `apoc.static.get()` function.

Putting everything together the Cypher query to search for podcasts using the Podcast Index API endpoint looks like this:

```cypher
WITH toString(timestamp()/1000) AS timestamp
WITH {
  `User-Agent`: "GRANDstackFM",
  `X-Auth-Date`: timestamp,
  `X-Auth-Key`: apoc.static.get('podcastkey'),
  `Authorization`: apoc.util.sha1([apoc.static.get('podcastkey') + apoc.static.get('podcastsecret') + timestamp])
} AS headers
CALL apoc.load.jsonParams("https://api.podcastindex.org/api/1.0/search/byterm?q=" + apoc.text.urlencode($searchTerm), headers, '', '') YIELD value
RETURN value
```

# GraphQL API

Now that we're able to use Cypher to search for podcasts it's time to start building our GraphQL API. We'll use the `create-grandstack-app` command line tool to quickly create the skeleton for our GraphQL server application.

## `create-grandstack-app`

The [`create-grandstack-app` CLI](https://grandstack.io/docs/getting-started-grand-stack-starter/#create-grandstack-app) is a utility for creating GRANDstack applications based on the GRANDstack starter project. Here we'll use it to create the GraphQL server application, but not the frontend since we're not ready to start thinking about the client application yet.

```shell
npx create-grandstack-app grandstack.fm
```

This command will fetch the latest release of the GRANDstack Starter project and ask us a few questions about how to connect to our Neo4j database.

<Image
  src="/static/images/grandstack-podcast-app-podcast-search-graphql-api/cli.png"
  alt="create-grandstack-app CLI"
  width={2364}
  height={1888}
/>

We now have a functional GraphQL server application in the `grandcast.fm/` directory that can serve a GraphQL endpoint backed by our local Neo4j database. Since we selected  the `API-Only` option we don't yet have a frontend for the application.

## `@cypher` GraphQL Schema Directive

Now it's time to edit the GraphQL type definitions of our GraphQL server application. We want to:

* create a `Query` field `podcastSearch` that returns a list of `PodcastSearchResult` objects
* add the Cypher query we wrote above as a `@cypher` schema directive, defining the logic for this `podcastSearch` field
* the `podcastSearch` field will take a String argument `searchTerm` that will be passed as a Cypher parameter to the indicated Cypher query

We'll replace the template type definitions found in `grandstack.fm/api/src/schema.graphql` with the following:

```graphql
type Query {
  podcastSearch(searchTerm: String!): [PodcastSearchResult]
  @cypher(
    statement: """
    WITH toString(timestamp()/1000) AS timestamp
    WITH {
      `User-Agent`: "GRANDstackFM",
      `X-Auth-Date`: timestamp,
      `X-Auth-Key`: apoc.static.get('podcastkey'),
      `Authorization`: apoc.util.sha1([apoc.static.get('podcastkey')+apoc.static.get('podcastsecret') +timestamp])
    } AS headers
    CALL apoc.load.jsonParams("https://api.podcastindex.org/api/1.0/search/byterm?q=" + apoc.text.urlencode($searchTerm), headers, '', '') YIELD value
    UNWIND value.feeds AS feed
    RETURN {
     itunesId: feed.itunesId,
     title: feed.title,
     description: feed.description,
     feedURL: feed.url,
     artwork: feed.artwork,
     categories: apoc.map.values(feed.categories, keys(feed.categories))
    }
    """
  )
}

type PodcastSearchResult {
  itunesId: String
  title: String
  description: String
  feedURL: String
  artwork: String
  categories: [String]
}
```

Next, in `src/index.js` we'll add the `PodcastSearchResult` to the types to be excluded from the schema augmentation process. Since this type represents results from the Podcast Index API call we don't want to generate query and mutation operations for this type. We'll also comment out the `init(driver)` line to initialize the database, saving that for later. 

We'll leave the rest of the file untouched:

```js
import { typeDefs } from './graphql-schema';
import { ApolloServer } from 'apollo-server-express';
import express from 'express';
import neo4j from 'neo4j-driver';
import { makeAugmentedSchema } from 'neo4j-graphql-js';
import dotenv from 'dotenv';
import { initializeDatabase } from './initialize';

// set environment variables from .env
dotenv.config();

const app = express();

/*
 * Create an executable GraphQL schema object from GraphQL type definitions
 * including autogenerated queries and mutations.
 * Optionally a config object can be included to specify which types to include
 * in generated queries and/or mutations. Read more in the docs:
 * https://grandstack.io/docs/neo4j-graphql-js-api.html#makeaugmentedschemaoptions-graphqlschema
 */

const schema = makeAugmentedSchema({
  typeDefs,
  config: {
    query: {
      exclude: ['PodcastSearchResult']
    },
    mutation: {
      exclude: ['PodcastSearchResult']
    }
  }
});

/*
 * Create a Neo4j driver instance to connect to the database
 * using credentials specified as environment variables
 * with fallback to defaults
 */
const driver = neo4j.driver(
  process.env.NEO4J_URI || 'bolt://localhost:7687',
  neo4j.auth.basic(
    process.env.NEO4J_USER || 'neo4j',
    process.env.NEO4J_PASSWORD || 'neo4j'
  ),
  {
    encrypted: process.env.NEO4J_ENCRYPTED ? 'ENCRYPTION_ON' : 'ENCRYPTION_OFF'
  }
);

/*
 * Perform any database initialization steps such as
 * creating constraints or ensuring indexes are online
 *
 */
const init = async (driver) => {
  await initializeDatabase(driver);
};

/*
 * We catch any errors that occur during initialization
 * to handle cases where we still want the API to start
 * regardless, such as running with a read only user.
 * In this case, ensure that any desired initialization steps
 * have occurred
 */

//init(driver)

/*
 * Create a new ApolloServer instance, serving the GraphQL schema
 * created using makeAugmentedSchema above and injecting the Neo4j driver
 * instance into the context object so it is available in the
 * generated resolvers to connect to the database.
 */
const server = new ApolloServer({
  context: { driver, neo4jDatabase: process.env.NEO4J_DATABASE },
  schema: schema,
  introspection: true,
  playground: true
});

// Specify host, port and path for GraphQL endpoint
const port = process.env.GRAPHQL_SERVER_PORT || 4001;
const path = process.env.GRAPHQL_SERVER_PATH || '/graphql';
const host = process.env.GRAPHQL_SERVER_HOST || '0.0.0.0';

/*
 * Optionally, apply Express middleware for authentication, etc
 * This also also allows us to specify a path for the GraphQL endpoint
 */
server.applyMiddleware({ app, path });

app.listen({ host, port, path }, () => {
  console.log(`GraphQL server ready at http://${host}:${port}${path}`);
});
```

## Querying With GraphQL

We can now start our GraphQL server application and start searching for podcasts:

```shell
➜ npm run start
...
Successfully compiled 6 files with Babel (411ms).
GraphQL server ready at http://0.0.0.0:4001/graphql
```

Now our GraphQL server application is running locally at `localhost:4001/graphql`. If we open up a web browser and navigate to that URL we'll see the GraphQL Playground tool. GraphQL Playground allows us to view the results of *introspection* of our GraphQL endpoint. This allows us to see the entry points for the GraphQL schema (the `Query`, `Mutation`, and `Subscription` fields) as well as the types, fields, and how the types are connected, essentially the schema of the API.

Let's query the GraphQL endpoint to find podcasts using the search term "Neo4j":

```graphql
{
  podcastSearch(searchTerm: "Neo4j") {
    title
    description
    artwork
    feedURL
    itunesId
    categories
  }
}
```

<Image
  src="/static/images/grandstack-podcast-app-podcast-search-graphql-api/query.png"
  alt="Querying the GraphQL endpoint with GraphQL Playground"
  width={2658}
  height={1036}
/>

We now have a GraphQL API to enable our users to search for podcasts. Next, we'll need to enable users to subscribe to podcasts and for that to work we'll need to think about how to add users and enable them to log in and authenticate. So that's what we'll focus on next time.

# Resources

- [Code on Github](https://github.com/johnymontana/grandcast.fm)
- [Neo4j YouTube Channel](https://www.youtube.com/neo4j)
- [Building a GRANDstack Real Estate search application](https://github.com/johnymontana/willow-grandstack)
- [Building a travel guide with Gatsby.js, GraphQL, & Neo4j](https://github.com/johnymontana/central-perk)
- [Arrows graph diagramming tool](http://www.apcjones.com/arrows/)
- [Podcast Index API](https://podcastindex.org/)
- [Fullstack GraphQL Application With GRANDstack book](https://grandstack.io/ebook)


Building A GRANDstack Podcast App: Episode 1

Podcast Search GraphQL API With Neo4j And The Podcast Index


import Image from 'next/image';

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/Dashboard.png"
  alt="Election night dashboard"
  height={1080}
  width={1920}
/>

A few weeks ago there was an election in the US. On election night as I was waiting for the results to come in I poured myself a stiff drink and sat down to [watch EJ Fox and Ian Johnson on the Coding With Fire livestream](https://observablehq.com/@codingwithfire) as they built data visualizations to help interpret the early returns data coming in. I thought it might be fun to try to import live election results data into Neo4j and see if I could make sense of the early returns. Fortunately the New York Times makes this easy enough by exposing a JSON API endpoint that returns live election returns. EJ and Ian pointed out this endpoint on the livestream and I thought I'd see what I could build using Neo4j tooling to help interpret and visualize the results.

## The Data

The JSON endpoint provided by the New York Times is [here.](http://static01.nyt.com/elections-assets/2020/data/api/2020-11-03/national-map-page/national/president.json) This endpoint returns a JSON object that includes election returns at the county level for each candidate.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/json.png"
  alt="JSON data"
  height={1952}
  width={2054}
/>

We can make use of the `apoc.load.json` procedure in the APOC Neo4j standard library to import this data into Neo4j by pulling this data directly from the endpoint and writing to Neo4j.

```cypher
CALL apoc.load.json("http://static01.nyt.com/elections-assets/2020/data/api/2020-11-03/national-map-page/national/president.json") YIELD value
UNWIND value.data.races AS race
MERGE (s:State {name: race.state_name})
SET s.electoral_votes = race.electoral_votes,
s.result = race.result,
s.id = race.state_id,
s.votes =race.votes,
s.absentee_votes = race.absentee_votes

WITH race,s
UNWIND race.counties AS county
MERGE (c:County {fips: county.fips})
SET c.name = county.name,
    c.trump = county.results.trumpd + county.results_absentee.trumpd,
    c.biden = county.results.bidenj + county.results_absentee.bidenj,
    c.expected_votes = county.tot_exp_vote,
    c.trump_lead = c.trump > c.biden,
    c.biden_lead = c.biden > c.trump,
    c.votes_2016 = county.votes2016,
    c.margin_2016 = county.margin2016
MERGE (c)-[:IN_STATE]->(s)
WITH null AS foobar
MATCH (s:State {result: "winner"})<-[:IN_STATE]-(c:County)
WITH s, sum(c.trump) AS trump, sum(c.biden) as biden
SET s.trump_winner = trump > biden,
    s.biden_winner = biden > trump
```

This Cypher script imports the data into Neo4j using a simple graph model of counties and states:

`(:County)-[:IN_STATE]->(:State)`

where we store properties such as the number of electoral votes on the `State` nodes, and votes by candidate on each `County` node.

### Periodic Data Updates

Since this data is coming in live on election night and updated frequently we want to make sure that we're regularly refreshing the data imported in Neo4j. We'll make use of another APOC procedure to do this: `apoc.periodic.schedule` which allows us to [schedule background jobs using Cypher.](https://neo4j.com/labs/apoc/4.1/background-operations/periodic-background/) Let's create a background job to update our data every 5 minutes:

```cypher
CALL apoc.periodic.schedule('election-import', statement, 300)
```

Here `statement` is the import statement above. We can make sure our background job is scheduled and running by listing all our background jobs:

```cypher
CALL apoc.periodic.list()
```

Now that we're importing and updating live election result data, it's time to start interpreting the results.

## Data Visualization With Neo4j Bloom

First, we'll use Neo4j Bloom to explore the data using an interactive graph visualization. Bloom is a super powerful graph data visualization tool for exploring data in Neo4j. Bloom allows us to express complex graph patterns to visualize using simple natural language search, meaning we don't need to write any Cypher to visualize our data. WebGL acceleration enables us to visualize and work with large graphs. Bloom is available in Neo4j Desktop so we just need to select "Neo4j Bloom" under the "Open in" dropdown for our database in Neo4j Desktop. You can learn more about Neo4j Bloom [here.](https://neo4j.com/bloom/)

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/bloom1.png"
  alt="Exploring the dataset with Neo4j Bloom"
  height={2272}
  width={3406}
/>

### Putting Things In Perspective

Graph visualizations in Bloom can be styled using a _perspective_. The perspective is a configuration that defines a certain business view or domain that defines how nodes and relationships should appear in the visualization. You can read more about perspectives in [the Neo4j Bloom documentation.](https://neo4j.com/docs/bloom-user-guide/current/bloom-perspectives/) We'll make use of _rule-based styling_ in our perspective to bind node size and color to our data to help convey election result details.

### Rule Based Styling

**Styling node size**

Rule-based styling allows us to bind node and relationship size and colors to our data. Let's style county nodes based on population size so that counties with larger population sizes appear as larger nodes.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/range1.png"
  alt="Configuring node size using range-based styling"
  height={1352}
  width={1900}
/>

Note that we need to specify the min and max values when creating a rule-based style configuration. We can run the following Cypher query to inspect our data and return the relevant min and max values:

```cypher
MATCH (c:County)
RETURN min(c.expected_votes), max(c.expected_votes)
```

Next, let's style state size according to the number of electoral votes allocated to each state. This means that states with more electoral votes (or more significance) will appear larger.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/range2.png"
  alt="Configuring state node size using range based styling"
  height={1536}
  width={1900}
/>

**Styling node color**

Similarly, we can also use rule-based styling to bind the color of the nodes in our visualization to our data. A typical use case here is to color nodes according to the result of a community detection or clustering algorithm. In this case we will add a rule to color County and State nodes blue in the case that the Democratic party candidate is leading and red for the jurisdictions where the Republican candidate is in the lead.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/range3.png"
  alt="Styling node color using rule-based styling"
  height={1556}
  width={1896}
/>

Now, as we explore the graph we can make use of the rule-based styles to quickly see the most significant jurisdictions and which way the early election returns are leaning in those jurisdictions.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/bloom2.png"
  alt="Using Neo4j Bloom"
  height={2272}
  width={3406}
/>

### Hierarchical Layout

A force directed layout is typically used in graph visualizations. This layout often results in groups of nodes that are more connected than others being grouped together. This allows us to visually interpret node clusters. However, sometimes the data we are working with is inherently hierarchical, such as a reporting hierarchy within an organization or in our case, a hierarchy of states and counties. Neo4j Bloom allows us to choose between force directed and hierarchical layouts when rendering the visualization.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/bloom3.png"
  alt="Neo4j Bloom hierarchical layout"
  height={2320}
  width={3406}
/>

And zoomed out the hierarchical layout even looks a bit like a Christmas Tree 🎄

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/bloom.png"
  alt="Neo4j Bloom hierarchical layout"
  height={2418}
  width={3808}
/>

## Building Dashboards With Neo4j Charts

Neo4j Bloom is great for interactive graph visualizations, however sometimes the answer to our question is best expressed in tabular form which can be visualized as charts, instead of graphs. Fortunately, [Adam Cowley](https://twitter.com/adamcowley) has recently released the Neo4j Charts graph app for Neo4j Desktop that allows us to build dashboards of charts to help interpret our graph data. For a full overview of the features of the Charts graph app see [this blog post.](https://medium.com/neo4j/creating-charts-from-your-graphs-2f5b4e86fd6c) First, we'll install the Charts graph app using the Graph App Gallery in Neo4j Desktop, or we can also visit [install.graphapp.io](https://install.graphapp.io)

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/install1.png"
  alt="The Graph App Gallery"
  height={1868}
  width={2482}
/>

After installing and opening the Charts graph app, let's get started creating a dashboard and adding reports to it. For each report that we add to the dashboard we'll need to include a Cypher query that fetches the data from Neo4j necessary to render the report chart. Let's start off with a simple "metric", showing a single value for the report, in this case the number of electoral votes assigned to each candidate. Our Cypher queries for this report match on each state then sum the number of electoral votes where the given candidate has been declared the winner:

```cypher
MATCH (s:State) WHERE s.biden_winner
RETURN SUM(s.electoral_votes)
```

and

```cypher
MATCH (s:State) WHERE s.trump_winner
RETURN SUM(s.electoral_votes)
```

And now let's add a metric report to our dashboard using this Cypher query.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/createchart2.png"
  alt="Adding a metric chart"
  height={2272}
  width={2784}
/>

Next, let's find counties that have higher than expected voter turnout. To do this we'll use the number of "expected votes" included in the NYT dataset and the number of votes reported so far per county to calculate a metric showing where the number of votes cast exceeds the expected number. Our query will return both the number of expected votes and total votes per county, ordered by the percentage of expected votes exceeded.

```cypher
MATCH (s:State)<-[:IN_STATE]-(c:County) WITH c.expected_votes AS expected_votes, c.biden + c.trump AS total_votes, c.name + ", " + s.id AS county
WITH toFloat(total_votes) / expected_votes AS turnout, county, expected_votes, total_votes
WITH * where turnout IS NOT NULL
RETURN "expected votes" AS key, expected_votes AS value, county AS index ORDER BY turnout DESC LIMIT 10

UNION

MATCH (s:State)<-[:IN_STATE]-(c:County) WITH c.expected_votes AS expected_votes, c.biden + c.trump AS total_votes, c.name + ", " + s.id AS county
WITH toFloat(total_votes) / expected_votes AS turnout, county, expected_votes, total_votes
WITH * where turnout IS NOT NULL
RETURN "total votes" AS key, total_votes AS value, county AS index ORDER BY turnout DESC LIMIT 10
```

We'll add this query to our dashboard by creating a stacked bar chart report.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/barchart.png"
  alt="Creating a bar chart"
  height={620}
  width={2554}
/>

Next, we want to determine which counties voted most overwhelmingly for a specific candidate. To do this we'll query for the counties with the highest percentage of votes for Biden:

```cypher
MATCH (s:State)<-[:IN_STATE]-(c:County) WHERE c.biden > 0 AND c.trump > 0
WITH (toFloat(c.biden) / (c.biden + c.trump)) AS pct_biden, c.name + ", " + s.id AS county
RETURN *  ORDER BY pct_biden DESC LIMIT 10
```

And the same for Trump:

```cypher
MATCH (s:State)<-[:IN_STATE]-(c:County) WHERE c.biden > 0 AND c.trump > 0
WITH (toFloat(c.trump) / (c.biden + c.trump)) AS pct_trump, c.name + ", " + s.id AS county
RETURN *  ORDER BY pct_trump DESC LIMIT 10
```

Now, let's add these queries to the dashboard using a line chart report:

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/linechart.png"
  alt="Adding a line chart"
  height={1286}
  width={2572}
/>

If we view our dashboard now we'll see all our report visualizations together.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/charts1.png"
  alt="Neo4j Charts graph app"
  height={2024}
  width={2686}
/>

We can add additional reports taking advantage of more complex charts such as radar charts, bump charts, and more. What interesting questions can you think of that can best be explained with these types of charts?

## Visualizing Geospatial Data With Neomap

Our election results dashboard wouldn't be complete without a map visualization. To create our map visualization we'll use another graph app for Neo4j Desktop called Neomap. Neomap was created by [Estelle Scifo](https://twitter.com/st3llasia) and allows us to visualize geospatial data from Neo4j. You can learn more about Neomap in this [introductory blog post](https://medium.com/neo4j/introducing-neomap-a-neo4j-desktop-application-for-spatial-data-3e14aad59db2) and an [example of visualizing routes with Neomap.](https://medium.com/neo4j/visualizing-shortest-paths-with-neomap-0-4-0-and-the-neo4j-graph-data-science-plugin-18db92f680de) We'll install Neomap using the Graph App Gallery in Neo4j Desktop or by visiting [install.graphapp.io](https://install.graphapp.io)

First, however we need to add some geospatial feature data to our database. There are a few options here: we could add polygon geometries for each state or county to create a choropleth map, but let's keep things simple and create a single marker for each county and color the marker according to the party of the candidate leading the jurisdiction (using the convention of red for Republican and blue for Democratic).

For these markers we'll need the latitude and longitude of the centroid of each county. I found [a dataset from the National Weather Service](https://www.weather.gov/gis/Counties) that includes the centroid of each county in the US.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/countydata.png"
  alt="Metadata for US county dataset"
  height={414}
  width={1378}
/>

This data is available as a shapefile, which is a binary format for geospatial data. We'll need to convert this to a format such as geojson or CSV which makes it easier to import into Neo4j. To do this we could use a tool like [GDAL](https://gdal.org/) which can convert geospatial data between just about every format, but today we'll use an online tool called [Mapshaper](https://mapshaper.org/) to convert this shapefile data to CSV. We'll open the shapefile in Mapshaper, select export to CSV and now we have our county centroid data in CSV format.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/mapshaper.png"
  alt="Mapshaper online geospatial data conversion tool"
  height={1494}
  width={2620}
/>

Now, let's use Neo4j's `LOAD CSV` functionality to import this data into our dataset. First, let's make sure we can read the CSV file and see what data we're working with:

```cypher
LOAD CSV WITH HEADERS FROM "file:///counties.csv" AS row
RETURN row LIMIT 1
```

```
{
  "FE_AREA": "se",
  "TIME_ZONE": "E",
  "FIPS": "23029",
  "COUNTYNAME": "Washington",
  "STATE": "ME",
  "LON": "-67.6361",
  "CWA": "CAR",
  "LAT": "45.0363"
}
```

We'll use the `FIPS` property to look up our County nodes and add a Point property to each county which represents the latitude and longitude of the centroid of each county.

```cypher
LOAD CSV WITH HEADERS FROM "file:///counties.csv" AS row
MATCH (c:County {fips: row.FIPS})
SET c.location = Point({latitude: toFloat(row.LAT), longitude: toFloat(row.LON)})
```

Now that we have our geospatial data for each county, in Neomap we'll add a layer to visualize the county data. With Neomap we can define a custom Cypher query returning the data we want to visualize, so we'll filter for counties where the Democratic candidate is leading and color the markers in that layer blue (repeating the process for red makers for the Republican candidate)

```cypher
MATCH (c:County) WHERE EXISTS(c.location) AND c.biden_lead
RETURN c.location.latitude AS latitude, c.location.longitude AS longitude
```

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/neomap1.png"
  alt="The Neomap graph app"
  height={1980}
  width={2732}
/>

When both layers are added we can see the distribution of each party's leading candidate across counties in the US. As a next step we might want to scale the size of each county marker according to population or perhaps render a polygon of the bounds of each county instead of a single marker, but this is a good start for our dashboard.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/map1.png"
  alt="Election map created using Neomap"
  height={822}
  width={1546}
/>

## The Dashboard

Putting everything together we now have an interactive live-updating election night dashboard showing graph data expressed in charts, graph visualization, and map view.

<Image
  src="/static/images/election-night-dashboard-neo4j-charts-bloom-visualization/Dashboard.png"
  alt="Election night dashboard"
  height={1080}
  width={1920}
/>


Using low code graph apps to build data visualizations with Neo4j.

Building An Election Night Dashboard With Neo4j Graph Apps: Bloom, Charts, And Neomap

Using the new Jolt format with HTTP in Neo4j 4.2 with edge network handlers.

Building A Location Aware Endpoint Using Neo4j And Cloudflare Workers



This post explores how to build spatial data visualizations using address data from the Paradise Papers leak of offshore corporations and the people connected to them. First, we geocode all addresses in the leaked data, then build a heatmap and interactive map for exploring the data of offshore legal entities.

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/heatmap1.png)

The [Paradise Papers](https://www.icij.org/investigations/paradise-papers/) is the latest data journalism investigation from the [International Consortium of Investigative Journalists (ICIJ)](https://www.icij.org). Like anyone working with complex and highly connected data (think of a chain of ownership of offshore legal entities) the ICIJ use [Neo4j](https://neo4j.com/) to find connections in the data. You can read more about how graph databases and Neo4j were used by the ICIJ for the Paradise Papers investigation [here](https://neo4j.com/blog/analyzing-paradise-papers-neo4j/) and [here](https://neo4j.com/blog/depth-graph-analysis-paradise-papers/).

## The Data

Anyone can download the Paradise Papers database as a Neo4j database from the [ICIJ's Offshore Leaks site](https://offshoreleaks.icij.org/pages/database). *Note that this database includes data from a number of ICIJ investigations including the Panama Papers, Paradise Papers, Bahamas Leaks and Offshore Leaks.*

Here's what the data model looks like:

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/datamodel.jpg)

As you can see each `Officer`, `Entity`, and `Intermediary` node is connected to an `Address` node through a `REGISTERED_ADDRESS` relationship. In the case of the Paradise Papers dataset, this address was found in the leaked documents or corporate registry.

I'm particularly interested in analyzing the data using this geographic component. We can use Cypher and Neo4j to ask questions of the data. For example, for Officers with an address in San Mateo, what are the most popular offshore jurisdictions for their legal entities?

```cypher
MATCH (a:Address)<-[:REGISTERED_ADDRESS]-(o:Officer)-[:CONNECTED_TO|OFFICER_OF]->(e:Entity)
WHERE a.name CONTAINS "San Mateo"
RETURN e.jurisdiction_description AS juris, COUNT(*) AS num
ORDER BY num DESC
```

```
╒════════════════╤═════╕
│"juris"         │"num"│
╞════════════════╪═════╡
│"Bermuda"       │90   │
├────────────────┼─────┤
│"Cayman Islands"│14   │
├────────────────┼─────┤
│"Mauritius"     │1    │
├────────────────┼─────┤
│"Seychelles"    │1    │
└────────────────┴─────┘
```

Since we have the addresses, we can use a process called geocoding to find the latitude and longitude for each address, enabling us to build spatial data visualizations for exploring the data.

## Geocoding Addresses

According to Wikipedia:

> Geocoding is the computational process of transforming a postal address description to a location on the Earth's surface (spatial representation in numerical coordinates)

There are several services that provide geocoding, including [Google's Geocoding API](https://developers.google.com/maps/documentation/geocoding/start) and [Open Street Maps' Nominatim](http://wiki.openstreetmap.org/wiki/Nominatim).

Thanks to Neo4j's spatial guru [Craig](https://twitter.com/craigtaverner) we can use both of these geocoding services directly from Cypher as part of the [APOC Cypher procedure library](
https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_geocode).

Once you've installed APOC there are a few settings to add to neo4j.conf:


```shell
dbms.security.procedures.unrestricted=apoc.*


apoc.spatial.geocode.provider=google
apoc.spatial.geocode.osm.throttle=1000 # be sure to throttle if using the free OSM API
apoc.spatial.geocode.google.throttle=0 # no need to throttle if using the PAID Google API
apoc.spatial.geocode.google.key=YOUR_GOOGLE_API_KEY_HERE
```


You can geocode a single address using the `apoc.spatial.geocodeOnce` procedure:

```cypher
CALL apoc.spatial.geocodeOnce('901 Trophy Hills Drive; Las Vegas; Nevada  89134; United States of America')
```


```json
{
  "formatted_address": "901 Trophy Hills Dr, Las Vegas, NV 89134, USA",
  "geometry": {
    "viewport": {
      "southwest": {
        "lng": -115.2948439802915,
        "lat": 36.1811390197085
      },
      "northeast": {
        "lng": -115.2921460197085,
        "lat": 36.1838369802915
      }
    },
    "location": {
      "lng": -115.293495,
      "lat": 36.182488
    },
    "location_type": "ROOFTOP"
  },
  "types": [
    "street_address"
  ],
  "address_components": [
    {
      "long_name": "901",
      "short_name": "901",
      "types": [
        "street_number"
      ]
    },
    {
      "long_name": "Trophy Hills Drive",
      "short_name": "Trophy Hills Dr",
      "types": [
        "route"
      ]
    },
    {
      "long_name": "Summerlin",
      "short_name": "Summerlin",
      "types": [
        "neighborhood",
        "political"
      ]
    },
    {
      "long_name": "Las Vegas",
      "short_name": "Las Vegas",
      "types": [
        "locality",
        "political"
      ]
    },
    {
      "long_name": "Clark County",
      "short_name": "Clark County",
      "types": [
        "administrative_area_level_2",
        "political"
      ]
    },
    {
      "long_name": "Nevada",
      "short_name": "NV",
      "types": [
        "administrative_area_level_1",
        "political"
      ]
    },
    {
      "long_name": "United States",
      "short_name": "US",
      "types": [
        "country",
        "political"
      ]
    },
    {
      "long_name": "89134",
      "short_name": "89134",
      "types": [
        "postal_code"
      ]
    },
    {
      "long_name": "0567",
      "short_name": "0567",
      "types": [
        "postal_code_suffix"
      ]
    }
  ],
  "place_id": "ChIJ412KJ_K_yIARxCzkI0IN7Ho"
}
```


We're only interested in the latitude and longitude, which we can grab from the `location` column:

```cypher
CALL apoc.spatial.geocodeOnce('901 Trophy Hills Drive; Las Vegas; Nevada  89134; United States of America') YIELD location
```

```json
{
  "description": "901 Trophy Hills Dr, Las Vegas, NV 89134, USA",
  "latitude": 36.182488,
  "longitude": -115.293495
}
```


So what we want to do is iterate through all the `Address` nodes in the Paradise Papers dataset, send the address text to the geocoding service and store three additional properties on the `Address` node: `latitude`, `longitude`, and `description`:

```cypher
MATCH (a:Address) WITH a LIMIT 1
CALL apoc.spatial.geocodeOnce(a.name) YIELD location
WITH a, location.latitude AS latitude, location.longitude AS longitude,
  location.description AS description
SET a.latitude = latitude,
    a.longitude = longitude,
    a.description = description
```

This works, but we have ~65k addresses to process and we're making one request at a time here. This results in about 1 request per second.


If we are using the paid Google geocoding API then we have a rate limit of 50 requests/second, so let's take advantage of that!

We can use [`apoc.periodic.iterate`](https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_apoc_periodic_iterate) to parallelize our requests. `periodic.iterate` takes two Cypher queries, runs the first query passing the results to the second query, which is then run in batches. Optionally, we can choose to run those batches in parallel.

**NOTE: This approach should only be used for the paid Google goeocoding API, otherwise you will exceed rate limits!**

```cypher
CALL apoc.periodic.iterate('MATCH (a:Address) RETURN a',
'CALL apoc.spatial.geocodeOnce(a.name) YIELD location
  WITH a, location.latitude AS latitude, location.longitude AS longitude,
  location.description AS description
SET a.latitude = latitude,
    a.longitude = longitude,
    a.description = description', {batchSize:100, parallel:true})
```


By executing our geocode procedure in parallel batches we can increase our throughput, significantly:

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/throughput1.png)


Once we've geocoded our addresses we can do things like calculate distance using Cypher.

*What are all the addresses in the dataset within 2km, and what offshore legal entities are they associated with?*

```cypher
MATCH (e:Entity)<-[:CONNECTED_TO|OFFICER_OF]-(o:Officer)-[:REGISTERED_ADDRESS]->(a:Address)
WHERE exists(a.latitude) AND exists(a.longitude)
AND DISTANCE(POINT(a), POINT({latitude: 37.5634399, longitude:-122.3244533})) < 2000
RETURN DISTINCT e.name AS offshore_entity, e.jurisdiction_description AS jurisdiction, a.name AS officer_address
```

**Query Results:**

| offshore_entity | jurisdiction | officer_address |
|---|---|---|
| AutoChina International Limited | Cayman Islands | 605 Hobart Ave; San Mateo; 94402 California; United States of America |
| Mill River Re Limited | Bermuda | 160 Bovet Road; San Mateo; CA 94402; United States of America |
| Indigo Reinsurance Ltd. | Bermuda | 160 Bovet Road; San Mateo; CA 94402; United States of America |
| Asurion Japan Reinsurance Ltd. | Bermuda | 160 Bovet Road; San Mateo; CA 94402; United States of America |
| International Wireless Communications Latin America Holdings Ltd. | Bermuda | 400 S El Camino Real; Suite 580; San Mateo; CA 94402; United States of America |
| Renters Reinsurance Company II, Ltd. | Bermuda | 108 Chukker Court; San Mateo CA 94403; United States of America |
| D&H Reinsurance Company, Ltd. | Bermuda | 108 Chukker Court; San Mateo CA 94403; United States of America |
| Farallon Reinsurance Company, Ltd. | Bermuda | 108 Chukker Court; San Mateo CA 94403; United States of America |
| H&F Rose Investors Ltd. | Bermuda | 133 Ridgeway Road; Hillsborough California 94010; United States of America |
| H&F International Rose Investors Ltd. | Bermuda | 133 Ridgeway Road; Hillsborough California 94010; United States of America |
| H&F Corporate Investors IV (Bermuda) Ltd. | Bermuda | 133 Ridgeway Road; Hillsborough California 94010; United States of America |
| Celtic Pharma Development Services Bermuda Ltd. | Bermuda | 109 Chukker Ct; San Mateo; CA 94403; United States of America |
| Mill River Re Limited | Bermuda | 1700 South El Camino Real; Suite 200; San Mateo; California 94402; United States of America |
| Indigo Reinsurance Ltd. | Bermuda | 1700 South El Camino Real; Suite 200; San Mateo; California 94402; United States of America |


but we can also build really cool interactive map based visualizations with our data!

## Building An Interactive Geographical Data Visualization

Using Leaflet.js to build an interactive geographic visualization, we want to pull data from Neo4j and annotate a map. To do this we'll make use of Leaflet.js (and a couple of plugins!), and the Javascript driver for Neo4j.

### Leaflet.js

[Leaflet.js](http://leafletjs.com/) is an open-source JavaScript library for creating interactive maps. We can create a simple map like this:

```html
<html>
<head>
  <style>
    #map {height: 100%;}
  </style>

  <script src="http://cdn.leafletjs.com/leaflet-0.4.4/leaflet.js"></script>
  <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.4.4/leaflet.css" />
</head>

  <body>
    <div id="map"></div>

    <script>
      var baseLayer = L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png',
        { attribution: 'OpenStreetMap', minZoom: 2, maxZoom: 18});

      var map = new L.Map('map', {center: new L.LatLng(37.5634399, -122.3244533), zoom: 17, layers: [baseLayer]});
    </script>

  </body>
</html>
```

This gives us a map:

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/map1.png)

now let's add some data to it! Leaflet allows us to create multiple layers and show/hide various layers. So we create a base map layer (our map tiles) and our data visualizations live in map layers that are plotted on top of the map tiles.

We'll make use of two types of geographical data visualizations - a heat map and marker clusters.


### Marker Clusters

Adding markers to our map is very simple in Leaflet, but if we add a marker for each address in the Paradise Papers dataset our map will soon look just like a pin cushion. Fortunately, the [MarkerCluster plugin for Leaflet](https://github.com/Leaflet/Leaflet.markercluster) includes logic for clustering markers together, depending on zoom level. Here's an example of how we can add a few markers to our Leaflet map:


```js

var markerLayer = new L.MarkerClusterGroup();

var marker1 = new L.Marker(new L.LatLng(37.5634399, -122.3222646));
marker1.bindPopup('Neo4j HQ');

var marker2 = new L.Marker(new L.LatLng(37.5669493, -122.3237035));
marker2.bindPopup('Philz Coffee');

var marker3 = new L.Marker(new L.LatLng(37.5622308, -122.3268279));
marker3.bindPopup('San Mateo Public Library');

markerLayer.addLayer(marker1);
markerLayer.addLayer(marker2);
markerLayer.addLayer(marker3);
map.addLayer(markerLayer);
```

Our markers are clustered together when zoomed out:

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/map2.png)

But once we zoom in we can see each individual marker:

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/map3.png)

### Heatmap

A heatmap is a data visualization (often imposed on a map, but could be done on a matrix as well) where colors are used to represent data values. In a geographical data visualization, where data points are often sparse, some form of interpolation is often used.

There are a [few different Leaflet heatmap plugins](http://leafletjs.com/plugins.html#heatmaps ) we can choose from but I like [heatcanvas](https://github.com/sunng87/heatcanvas) as its API provides several configuration options for configuring the formula used to determine the value of a pixel.

Here's how we can add a simple heatmap to our Leaflet map using heatcanvas:



```js
var heatmap = new L.TileLayer.HeatCanvas({}, {'step':0.3, 'degree': HeatCanvas.LINEAR, 'opacity': 0.7});
heatmap.pushData(37.5634399, -122.3222646, 10.0); // Neo4j HQ
heatmap.pushData(37.5669493, -122.3237035, 20.0); // Philz Coffee
heatmap.pushData(37.5622308, -122.3268279, 12.0); // Public Library
map.addLayer(heatmap);
```

![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/map4.png)
*And our heatmap shows that I spend much more time at the coffeeshop and library than I do at work ;-)*

### Using Data From Neo4j

Now that we've seen how to create a map using Leaflet, adding markers, marker clusters, and a heatmap let's see how we can pull data out of Neo4j to populate our map. Since we previously added `latitude` and `longitude` properties to our `Address` nodes we'll use those to mark points on our map, but we'd also like to include information about the `Officer` connected to that address and the legal entities the officer is connected to. This Cypher query will find all registered addresses of Officers in the Paradise Papers dataset (that we were able to geocode), as well as all their connected offshore legal entities:


```cypher
MATCH (a:Address)<-[:REGISTERED_ADDRESS]-(o:Officer)-[:CONNECTED_TO|OFFICER_OF]-(e:Entity)
  WHERE exists(a.latitude) and exists(a.longitude)
RETURN a.name AS address, a.latitude AS latitude, a.longitude AS longitude,
  COLLECT(DISTINCT o.name) AS officers, COLLECT(DISTINCT e.name) AS entities,
  COLLECT(DISTINCT e.jurisdiction_description) AS jurisdictions, 1.0*COUNT(*) AS strength
```

We can make use of [the JavaScript driver for Neo4j](https://github.com/neo4j/neo4j-javascript-driver) to run this query, and add data to the map for each record returned:

```js

var driver = neo4j.v1.driver("bolt://localhost:7687", neo4j.v1.auth.basic("neo4j", "MYPASSWORD"));
var session = driver.session();
session
  .run(`MATCH (a:Address)<-[:REGISTERED_ADDRESS]-(o:Officer)-[:CONNECTED_TO|OFFICER_OF]-(e:Entity)
        WHERE exists(a.latitude) and exists(a.longitude)
        RETURN a.name AS address, a.latitude AS latitude, a.longitude AS longitude,
          COLLECT(DISTINCT o.name) AS officers, COLLECT(DISTINCT e.name) AS entities,
          COLLECT(DISTINCT e.jurisdiction_description) AS jurisdictions, 1.0*COUNT(*) AS strength`)
  .subscribe({
    onNext: function (record) {
      //console.log(record);
      var marker = new L.Marker(new L.LatLng(record.get('latitude'), record.get('longitude')));
      marker.bindPopup('<b>Address:</b> ' + record.get('address') + '<br>' + '<b>Officers: </b>' + record.get('officers').toString() + '<br><b>Entities:</b> ' + record.get('entities').toString() + '<br><b>Offshore Countries: </b> ' + record.get('jurisdictions').toString());
      markerLayers.addLayer(marker);
      heatmap.pushData(record.get('latitude'), record.get('longitude'), record.get('strength')*0.01);
    },
    onCompleted: function () {
      var overlayMaps = {'Markers': markerLayers, 'Heatmap': heatmap};
      var controls = L.control.layers(null, overlayMaps, {collapsed: false, autoZIndex: true});
      map = new L.Map('map', {center: new L.LatLng(51.505, -0.09), zoom: 3, layers: [baseLayer]});
      map.addLayer(markerLayers);
      map.addLayer(heatmap);
      controls.addTo(map);
      session.close();
    },
    onError: function (error) {
      console.log(error);
    }
  });
```


Now we have an interactive geographic visualization of addresses from the Paradise Papers dataset that we can explore.

Both the heatmap, to show us areas that have a high concentration of addresses associated with officers of offshore entities:

[![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/heatmap1.png)](http://www.lyonwj.com/pp-viz/heatmap/)

And we can also explore the markers to find officers and connected offshore legal entities. Here we can see casino owner [Sheldon Adelson and his Bermuda-based company](https://projects.icij.org/paradise-papers/the-influencers/#/sheldon-adelson) used for registering private aircraft:

[![](/static/images/geocoding-paradise-papers-neo4j-spatial-visualization/marker.png)](http://www.lyonwj.com/pp-viz/heatmap/)


These type of geographical data visualizations are particularly useful for data journalists who are interested in finding stories relevant for a certain geographic area.


All code is available on [Github](https://github.com/johnymontana/pp-viz) and you can try the [live map visualization](http://www.lyonwj.com/pp-viz/heatmap/). You can [download the Paradise Papers Neo4j database from ICIJ](https://offshoreleaks.icij.org/pages/database), or try it as a [Neo4j Sandbox.](https://neo4j.com/sandbox-v2/)

This post explores how to build spatial data visualizations using address data from the Paradise Papers leak of offshore corporations and the people connected to them. First, we geocode all addresses in the leaked data, then build a heatmap and interactive map for exploring the data of offshore legal entities.

Geocoding Paradise Papers Addresses In Neo4j To Build Interactive Geographical Data Visualizations


![](/static/images/entity-extraction-russian-troll-tweets-neo4j/louisiana.png)

# Russian Twitter Troll Tweets

[Previously](http://www.lyonwj.com/2017/11/12/scraping-russian-twitter-trolls-python-neo4j/), we explored how to scrape tweets from Internet Archive that have been removed from Twitter.com and the Twitter API as a result of the US House Intelligence Committee's investigation into Russia's involvement in influencing the 2016 election through social media, largely by spreading fake news.

These accounts were identified by Twitter as connected to Russia's Internet Research Agency, a company believed to have been involved in spreading fake news in an attempt to influence the US election, however Twitter has removed all data related to these accounts.

Our previous post focused on scraping Internet Archive to retrieve the data and import into Neo4j. We also looked at some Cypher queries we could use to analyze the data. In this post we make use of a natural language processing technique called entity extraction to enrich our graph data model and help us explore the dataset of Russian Twitter Troll Tweets. For example, can we try to see what people, places, and organizations these accounts were tweeting about in the months leading up to the 2016 election?

# Named Entity Recognition

According to [Wikipedia](https://en.wikipedia.org/wiki/Named-entity_recognition):

> Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.

Entity extraction is an important task in natural language processing and information extraction and can be used to help make sense of large bodies of text.

## Entity Extraction With Polyglot

There are several NLP libraries and services available (such as [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/), [IBM's AlchemyAPI](https://www.ibm.com/watson/alchemy-api.html), [Thomson Reuters' Open Calais](http://www.opencalais.com/)) that offer entity extraction. One tool that I like is [Polyglot](https://github.com/aboSamoor/polyglot), a Python library and command line tool for NLP. Polyglot supports *many* different languages (up to 165 for some NLP tasks) and a wide range of NLP features including language detection, sentiment analysis, part of speech tagging, and word embeddings.

In this post we'll make use of Polyglot's entity extraction functionality to find entities mentioned in the tweets in our Russian Twitter Troll tweet data.

Other entity extraction tools use supervised learning approaches that require human labeled training data. Polyglot however uses link structure from Wikipedia and Freebase that results in a language agnostic technique for entity extraction. You can read more about their technique in this [paper.](https://sites.google.com/site/rmyeid/papers/polyglot-ner.pdf?attredirects=0&d=1)

## Installing Polyglot

From looking at some issues online it seems many folks struggle with installing Polyglot, especially on MacOS. These are the steps I used to install Polyglot with the Anaconda Python distribution on Ubuntu 16.04, including Jupyter, thanks to [this gist](https://gist.githubusercontent.com/linwoodc3/8704bbf6d1c6130dda02bbc28967a9e6/raw/786fc5074c811b3ec359135f2ce8d92d8413b33a/polyglotOnMacOSX35.sh).

```shell
conda create -n icutestenv python=3.5.2 --no-deps -y
source activate icutestenv
conda install -c ccordoba12 icu=54.1 -y
conda install ipython numpy scipy jupyter notebook -y
python -m ipykernel install --user
easy_install pyicu
pip install morfessor
pip install pycld2
pip install polyglot
```

# Applying Entity Extraction To The Russian Twitter Troll Dataset

Our approach will be to use the Python driver for Neo4j to retrieve all the tweet text in our dataset from Neo4j, run the Polyglot entity extraction algorithm on the text of each tweet, saving the tweet text, tweet id, and extracted entities to a JSON file that we can then use to update the Neo4j graph, extending the datamodel to connect `Tweet` nodes to new `Person`, `Location`, and `Organization` nodes as identified by Polyglot.

## Running Polyglot Entity Extraction

Polyglot entity extraction works by taking a piece of text (tweets in our case), and annotating (or tagging) the text where it recognizes named entities.

Polyglot can recognize three types of entities:

* **Locations** *(Tag: I-LOC)*
  * cities, countries, regions, continents, neighborhoods, etc.
* **Organizations** *(Tag: I-ORG)*:
  * companies, schools, newspapers, political parties, etc.
* **Persons** *(Tag: I-PER)*:
  * politicians, celebrities, authors, etc.

Here's a simple example:

```python
# simple Polyglot entity extraction example

from polyglot.text import Text
blob = Text('@realDonaldTrump "Hillary Clinton has zero record to run on - unless you call corruption positive.." - @IngrahamAngle')
blob.entities
```


And we can see that Polyglot recognized "Hillary Clinton" as a `Person` entity:

```md
[I-PER(['Hillary', 'Clinton'])]
```

Note that Polyglot did not tag either of the two Twitter screen names, `@realDonaldTrump` or `@IngramAngle` as entities. Since our data has Twitter mentions we can infer these already.

## Retrieving Tweet Data From Neo4j

We start with the Neo4j database that we built from scraped tweets in the [previous post](http://www.lyonwj.com/2017/11/12/scraping-russian-twitter-trolls-python-neo4j/). We want to query that database for all tweets, retrieving the text of the tweet and the tweet id. We create a Python dict for each tweet that contains the id and text and then append each tweet dict to a list:

```python
from neo4j.v1 import GraphDatabase
driver = GraphDatabase.driver("bolt://localhost:7687")


with driver.session() as session:
    results = session.run("MATCH (t:Tweet) RETURN t.text AS text, t.tweet_id AS tweet_id")

tweetObjArr = []

for r in results:
    tweetObj = {}
    tweetObj['id'] = r['tweet_id']
    tweetObj['text'] = r['text']
    tweetObjArr.append(tweetObj)

```

Next, we iterate through each tweet dict in the array and run the Polyglot entity extaction algorithm for each tweet. We create a new dict, including any tagged entities and append those to a new array:


```python
entityArr = []


for t in tweetObjArr:
    try:
        parsedTweet = {}
        parsedTweet['id'] = t['id']
        parsedTweet['text'] = t['text']
        blob = Text(t['text'])
        entities = blob.entities
        parsedTweet['entities'] = []
        for e in entities:
            eobj = {}
            eobj['tag'] = e.tag
            eobj['entity'] = e
            parsedTweet['entities'].append(eobj)
        if len(parsedTweet['entities']) > 0:
            entityArr.append(parsedTweet)
    except:
        pass
```

*Note that we could write this in a more pythonic list comprehension, but I think this version is better for seeing what exactly is going on.*


Here's an example of what we have at this point:

```json
{
        "entities": [
            {
                "entity": [
                    "Republicans"
                ],
                "tag": "I-ORG"
            },
            {
                "entity": [
                    "FBI"
                ],
                "tag": "I-ORG"
            },
            {
                "entity": [
                    "Clinton"
                ],
                "tag": "I-PER"
            }
        ],
        "id": 784163886547230720,
        "text": "RT @OffGridInThePNW: Republicans blast FBI for 'astonishing' agreement to destroy Clinton aides' laptops |  https://t.co/cwEUtK9i9V"
    }
```

Next, we save this array of entities and tweets to a json file that we'll use in the next step to import into Neo4j:

```python
import json
with open('parsed_tweets.json', 'w') as f:
    json.dump(entityArr, f, ensure_ascii=False, sort_keys=True, indent=4)
```


```md
{'entities': [{'entity': I-PER(['Hillary', 'Clinton']), 'tag': 'I-PER'}],
 'id': 773585101489922048,
 'text': '@realDonaldTrump "Hillary Clinton has zero record to run on - unless you call corruption positive.." - @IngrahamAngle'}
 ```


## Importing Extracted Entities Into Neo4j

So far our datamodel looks like this:

![](/static/images/entity-extraction-russian-troll-tweets-neo4j/datamodel_pre.png)

What we want to do is extend the data model to include `Person`, `Location`, and `Organization` nodes, connected to any tweets that contain references to these entities:

![](/static/images/entity-extraction-russian-troll-tweets-neo4j/datamodel_post.png)

From the previous step, we have a json file of tweet text, tweet id, and extracted entities:

```md
{'entities': [{'entity': ['California'], 'tag': 'I-LOC'},
              {'entity': ['Cleveland'], 'tag': 'I-LOC'},
              {'entity': ['Nicholas', 'Rowe'], 'tag': 'I-PER'}],
 'id': '669554555835682816',
 'text': 'California man killed in Cleveland was shot in the head and back: '
         'Nicholas Rowe was identified as the man found dead Nov. 20 b...  '
         '#crime'}
```

First, we execute three Cypher queries to add uniqueness constraints to our database. The uniqueness constaints allow us to assert that no duplicate nodes will be created. For example, we only want one node with the label `Location` and `name` property `California` to represent the state of California. Any tweets that contain "California" should all have a relationship to the canonical "California" node. Uniqueness constraints give us this guarantee. Uniqueness constraints also create an index, which will speed our import significantly.

```python
with driver.session() as session:
    session.run('CREATE CONSTRAINT ON (p:Person) ASSERT p.name IS UNIQUE;')
    session.run('CREATE CONSTRAINT ON (l:Location) ASSERT l.name IS UNIQUE;')
    session.run('CREATE CONSTRAINT ON (o:Organization) ASSERT o.name IS UNIQUE;')
```

Next, we load our json file that contains tweets and entities:

```python
with open("parsed_tweets.json") as f:
    parsed_tweets = json.load(f)
```

And define a Cypher query that will take this array of tweets and entities, iterate through them and update the graph, creating new `Person`, `Organization`, and `Location` nodes and creating relationships to any tweets that contains these entities:

```python
entity_import_query = '''
WITH $parsedTweets AS parsedTweets
UNWIND parsedTweets AS parsedTweet
MATCH (t:Tweet) WHERE t.tweet_id = parsedTweet.id


FOREACH(entity IN parsedTweet.entities |
    // Person
    FOREACH(_ IN CASE WHEN entity.tag = 'I-PER' THEN [1] ELSE [] END |
        MERGE (p:Person {name: reduce(s = "", x IN entity.entity | s + x + " ")}) //FIXME: trailing space
        MERGE (p)<-[:CONTAINS_ENTITY]-(t)
    )

    // Organization
    FOREACH(_ IN CASE WHEN entity.tag = 'I-ORG' THEN [1] ELSE [] END |
        MERGE (o:Organization {name: reduce(s = "", x IN entity.entity | s + x + " ")}) //FIXME: trailing space
        MERGE (o)<-[:CONTAINS_ENTITY]-(t)
    )

    // Location
    FOREACH(_ IN CASE WHEN entity.tag = 'I-LOC' THEN [1] ELSE [] END |
        MERGE (l:Location {name: reduce(s = "", x IN entity.entity | s + x + " ")}) // FIXME: trailing space
        MERGE (l)<-[:CONTAINS_ENTITY]-(t)
    )
)

'''
```

Note that we make use of the [`FOREACH( _ IN CASE WHEN ... THEN [1] ELSE [] END | ... )` trick](http://www.markhneedham.com/blog/2014/06/17/neo4j-load-csv-handling-conditionals/) for handling conditionals.

Finally, we use the Neo4j Python driver to run our import query, passing in the array of dicts as a parameter:

```python
with driver.session() as session:
    session.run(entity_import_query, parsedTweets=parsed_tweets)
```

## Making sense of our extracted entities

Now that we've enriched our graph model we can answer a few more interesting questions of our Russia Twitter Troll dataset.

### Most frequently mentioned entities

```cypher
// Most frequently mentioned Person entities
MATCH (t:Tweet)-[:CONTAINS_ENTITY]->(p:Person)
RETURN p.name AS name, COUNT(*) AS num ORDER BY num DESC LIMIT 10
```

```md
╒══════════════════╤═════╕
│"name"            │"num"│
╞══════════════════╪═════╡
│"Trump "          │22   │
├──────────────────┼─────┤
│"… "              │18   │
├──────────────────┼─────┤
│"Obama "          │15   │
├──────────────────┼─────┤
│"Clinton "        │12   │
├──────────────────┼─────┤
│"Hillary "        │12   │
├──────────────────┼─────┤
│"sanders "        │7    │
├──────────────────┼─────┤
│"Bernie Sanders " │6    │
├──────────────────┼─────┤
│"Donald Trump "   │6    │
├──────────────────┼─────┤
│"pic.twitter.com "│6    │
├──────────────────┼─────┤
│"Hillary Clinton "│6    │
└──────────────────┴─────┘
```

```cypher
// Most frequently mentioned Location entities
MATCH (t:Tweet)-[:CONTAINS_ENTITY]->(l:Location)
RETURN l.name AS name, COUNT(*) AS num ORDER BY num DESC LIMIT 10
```

```md
╒══════════════╤═════╕
│"name"        │"num"│
╞══════════════╪═════╡
│"Louisiana "  │13   │
├──────────────┼─────┤
│"Chicago "    │13   │
├──────────────┼─────┤
│"Cleveland "  │11   │
├──────────────┼─────┤
│"baltimore "  │9    │
├──────────────┼─────┤
│"Miami "      │8    │
├──────────────┼─────┤
│"Ohio "       │6    │
├──────────────┼─────┤
│"California " │6    │
├──────────────┼─────┤
│"Iraq "       │6    │
├──────────────┼─────┤
│"Westboro "   │5    │
├──────────────┼─────┤
│"New Orleans "│5    │
└──────────────┴─────┘
```


### Exploring Tweets Around Certain Entities


```cypher
// Tweets, hashtags, and entities around Louisiana
MATCH (u:User)-[:POSTED]->(t:Tweet)-[:CONTAINS_ENTITY]->(l:Location {name: "Louisiana "}  )
OPTIONAL MATCH (t)-[:HAS_TAG]->(ht:Hashtag)
OPTIONAL MATCH (t)-[:CONTAINS_ENTITY]->(e)
RETURN *
```

![](/static/images/entity-extraction-russian-troll-tweets-neo4j/louisiana.png)



You can find the code for this project on [Github](https://github.com/johnymontana/russian-twitter-trolls), in [this Jupyter notebook](https://github.com/johnymontana/russian-twitter-trolls/blob/master/import/polyglot.ipynb):


Natural language processing (NLP) techniques like entity extraction can be used to help make sense of a large text corpus. In this post we apply named entity resolution to the scraped Russian Twitter Troll tweets to try to get a better understanding of how these trolls were spreading fake news.

Applying NLP and Entity Extraction To The Russian Twitter Troll Tweets In Neo4j (and more Python!)

In this post we explore how to scrape tweets from Internet Archive for Russian Twitter Troll accounts, import into Neo4j for analysis, and how to build a simple GraphQL API exposing the data through GraphQL.

Scraping Russian Twitter Trolls With Python, Neo4j, and GraphQL

Many web services are converting their publicly facing APIs from REST to GraphQL. Companies like GitHub and Shopify have been leading this transition to GraphQL. In this post we take a look at how we can query the new Mattermark GraphQL API using Apollo Client, storing the results in Neo4j to then see what we can learn about a local startup ecosystem.

Analyzing A Local Startup Ecosystem With Mattermark, GraphQL, Apollo Client, and Neo4j



![](/static/images/trumpworld-us-contracting-data-neo4j/vornado-graph-trump.svg)

One of the powers of working with graph databases is the ability to combine disparate datasets and query across them. Today we'll look at how we can combine the BuzzFeed Trumpworld graph with data about federal government contracts from USASpending.gov, allowing us to examine any government contracts that were awarded to organizations that appear in Trumpworld.

# The BuzzFeed Trumpworld Graph

A few weeks ago [BuzzFeed released a dataset](https://www.buzzfeed.com/johntemplon/help-us-map-trumpworld/) of people and organizations with connections to Donald Trump, calling it the Trumpworld dataset. They wanted to make this data available to the public as part of their work, and also solicit feedback from the community to help find missing connections.


[![](/static/images/trumpworld-us-contracting-data-neo4j/buzzfeed-data.png)](https://www.buzzfeed.com/johntemplon/help-us-map-trumpworld/)
*The BuzzFeed Trumpworld dataset. People, organizations, and their connections around Donald Trump.*


My colleague [Michael Hunger](https://twitter.com/mesirii) wrote import scripts for loading this data into Neo4j. He also wrote [an excellent blog post about working with the Trumpworld Graph in Neo4j](https://neo4j.com/blog/buzzfeed-trumpworld-dataset-neo4j/). If you haven't read it yet, start there because this post takes his work and builds on it by integrating data on US government contracts.

By working with this data in Neo4j we can write queries, such as "Show me the second degree network of Jared Kushner":

```cypher
MATCH network = (:Person {name:"JARED KUSHNER"})-[*..2]-()
RETURN network
```

![](/static/images/trumpworld-us-contracting-data-neo4j/kushner-second-degree.png)
*The second degree network of Jared Kushner.*

# Adding US Government Contracting Data - USASpending.gov

Data on recipients of federal government contracts is available online, at [USASpending.gov](https://www.usaspending.gov/Pages/Default.aspx).

![](/static/images/trumpworld-us-contracting-data-neo4j/usaspending.png)
*Data download page for USASpending.gov. I started with contracts issued by all agencies for the 2016 fiscal year.*

We may need to reference the [data dictionary for USASpending.gov](https://www.usaspending.gov/DownloadCenter/Documents/USAspending.govDownloadsDataDictionary.pdf)

## The Data

Each row in the CSV file represents a contract that has been awarded to an organization, issued by a federal government agency (actually each row represents a *transaction*, and we could have multiple transactions per contract, but let's simplify our thinking and just think of each row as a contract).

Looking at the data dictionary we can see that each row has *a lot* of fields (225 actually!), so there is a lot of information encoded in each row. We are mainly interested in:

* What organization was awarded the contract?
* What is the parent organization of the awardee (is it a subsidiary)?
* The amount of the contract
* The agency issuing the contract
* The purpose of the contract

We can use Cypher's `LOAD CSV` functionality to exaimine the first few rows of the CSV file and return specifically the columns we are interested in:

```cypher
// Load the contractors CSV file and return the first 5 rows 
// but only the columns we're interested in
LOAD CSV WITH HEADERS 
FROM "file:///2016_All_Contracts_Full_20170115.csv" AS row
WITH row LIMIT 5
RETURN row.piid AS piid, 
       row.fiscal_year AS fiscal_year, 
       row.vendorname AS vendor_name, 
       row.mod_parent AS parent_org, 
       row.dollarsobligated AS amount, 
       row.contractingofficeagencyid AS agency, 
       row.productorservicecode AS purpose
```

![](/static/images/trumpworld-us-contracting-data-neo4j/trump-table-1.png)

I found a few rows where the quote escaping wasn't quite right, and the Neo4j CSV parser wasn't able to parse the row correctly. I removed any of these rows using `sed`. For example:

```shell
$sed -i '4671792 d' 2016_All_Contracts_Full_20170115.csv
```


So here we can see some contracts that were issued by the Federal Prison System to various vendors. Now we could certainly import the entire dataset into Neo4j and run some interesting queries for analysis, but we're specifically interested in finding contracts that were awarded to vendors that appear in the Trumpworld dataset. Let's see if we can find any. To do this we need to `MATCH` on `Organization` nodes where the `name` property of the organization is equal to the `vendorname` column in the CSV file. We have to look through 5000 rows before we find a match: 

```cypher

LOAD CSV WITH HEADERS FROM 
"file:///2016_All_Contracts_Full_20170115.csv" AS row
WITH row LIMIT 50000
MATCH (o:Organization) WHERE o.name = row.vendorname
RETURN row.piid AS piid, 
       row.fiscal_year AS fiscal_year, 
       row.vendorname AS vendor_name, 
       row.mod_parent AS parent_org, 
       row.dollarsobligated AS amount, 
       row.contractingofficeagencyid AS agency, 
       row.productorservicecode AS purpose
```

![](/static/images/trumpworld-us-contracting-data-neo4j/trump-table-2.png)


Here we see Duke University showing up as a vendor. This means that Duke University must be in the Trumpworld data. How is Duke University connected to Donald Trump?

```cypher
MATCH p=(duke:Organization)-[*1..5]-(donald:Person)
WHERE duke.name   = "DUKE UNIVERSITY" AND
      donald.name = "DONALD J. TRUMP"
RETURN p LIMIT 1
```

![](/static/images/trumpworld-us-contracting-data-neo4j/duke-univ.png)
*How is Donald Trump connected to Duke University?*


We can see that in this case the `vendor_name` and `parent_org` values are the same. This means that Duke University isn't part of larger organization. Parent company information is provided by Dun & Bradstreet. We are also interested in finding contracts that were awarded to subsidiaries of companies in Trumpworld, as this would also help us identify potential conflicts of interest. To find these contracts (one that was awarded to a vendor where the vendor is a subsidiary of a company that appears in Trumpworld) we'll need to match on `Organization` nodes where the `name` property is equal to the `parent_org` column and where the `vendor_name` is not equal to `parent_org`. To find examples of these we need to look through a few more rows - the first 50000 (but don't worry there are more than 4 million rows in this CSV file!):

```cypher
LOAD CSV WITH HEADERS 
FROM "file:///2016_All_Contracts_Full_20170115.csv" AS row
WITH row LIMIT 50000
MATCH (o:Organization) 
WHERE o.name = row.mod_parent 
    AND NOT row.mod_parent = row.vendorname
RETURN row.piid AS piid, 
       row.fiscal_year AS fiscal_year, 
       row.vendorname AS vendor_name, 
       row.mod_parent AS parent_org, 
       row.dollarsobligated AS amount, 
       row.contractingofficeagencyid AS agency, 
       row.productorservicecode AS purpose
```

![](/static/images/trumpworld-us-contracting-data-neo4j/trump-table-3.png)

## The Labeled Property Graph Model

How might we model this contract data as a graph? Let's examine the third contract in the table above:

![](/static/images/trumpworld-us-contracting-data-neo4j/trump-table-4.png)

![](/static/images/trumpworld-us-contracting-data-neo4j/contract-data-model.png)
*How should we model government contracts in Neo4j?*

The entities we want to model are `Organization`, `Contract`, and government `Agency`. 

Note that we have an overlap on `Organization`, which allows us to combine these two datasets and query across them.

## Loading The Data

Now we're ready to import the data into the Trumpworld graph. We'll end running two import queries:

**First, where the vendor of the contract appears as an organization in Trumpworld**

```cypher
LOAD CSV WITH HEADERS 
FROM "file:///2016_All_Contracts_Full_20170115.csv" AS row

MATCH (o:Organization) WHERE o.name = row.vendorname

WITH o,row.piid AS piid, 
     row.fiscal_year AS fiscal_year, 
     row.vendorname AS vendor_name, 
     row.mod_parent AS parent_org, 
     toFloat(row.dollarsobligated) AS amount, 
     substring(row.contractingofficeagencyid, 6) AS agency, 
     row.productorservicecode AS purpose

MERGE (a:Agency {name: agency})
MERGE (c:Contract {piid: piid})
ON CREATE SET c.amount = amount,
    c.purpose = purpose,
    c.fiscal_year = fiscal_year
// sum the transactions per contract
ON MATCH SET c.amount = c.amount + amount
MERGE (a)-[:ISSUED_CONTRACT]->(c)
MERGE (c)<-[:AWARDED_CONTRACT]-(o)
```

**Then, a second query for loading contracts where the vendor's *parent company* appears as an organization in Trumpworld**

```cypher
LOAD CSV WITH HEADERS 
FROM "file:///2016_All_Contracts_Full_20170115.csv" AS row

MATCH (parent:Organization) 
WHERE parent.name = row.mod_parent 
    AND NOT row.mod_parent = row.vendorname
WITH parent, row.piid AS piid, row.fiscal_year AS fiscal_year, 
    row.vendorname AS vendor_name, row.mod_parent AS parent_org, 
    toFloat(row.dollarsobligated) AS amount, 
    substring(row.contractingofficeagencyid, 6) AS agency, 
    row.productorservicecode AS purpose
MERGE (vo:Organization {name: vendor_name})
MERGE (a:Agency {name: agency})
MERGE (c:Contract {piid: piid})
ON CREATE SET c.amount = amount,
    c.purpose = purpose,
    c.fiscal_year = fiscal_year
// sum the transactions per contract
ON MATCH SET c.amount = c.amount + amount
MERGE (a)-[:ISSUED_CONTRACT]->(c)
MERGE (c)<-[:AWARDED_CONTRACT]-(vo)
MERGE (parent)<-[:PARENT_ORG]-(vo)
```


## Analyzing Trumpworld + USASpending

Now that we've imported the USASpending data, let's take a look at the Merchandise Mart Properties, Inc contract. Since this is the Trumpworld graph, a logical question we might ask is "What is the connection from this contract to Donald Trump?". We can do this using the `shortestPath` function in Cypher:

```cypher
MATCH (c:Contract {piid: "DJU4500517056"})
MATCH (c)<-[:ISSUED_CONTRACT]-(a:Agency)
MATCH (donald:Person {name: "DONALD J. TRUMP"})
MATCH p=shortestPath( (donald)-[*]-(c) )
RETURN *
```

![](/static/images/trumpworld-us-contracting-data-neo4j/vornado-graph-trump.svg)
*The shortest path from a specific contract to Donald Trump.*

Let's digest the result of this query. This graph visualization is showing us a contract awarded by the Federal Prison System for the lease of facilities that was awarded to a real estate company whose parent company is an investor in a company whose CEO is the son-in-law (and close advisor) of Donald Trump.

Why is this interesting? Is this a potential conflict of interest? I don't know, but I think it's pretty cool that we can find these kind of connectons using Neo4j and Cypher!

Of course, this is just one shortest path, there may be others. Let's find *all* shortest paths connecting this contract to Donald Trump:


```cypher
MATCH (c:Contract {piid: "DJU4500517056"})
MATCH (c)<-[:ISSUED_CONTRACT]-(a:Agency)
MATCH (donald:Person {name: "DONALD J. TRUMP"})
MATCH p=allShortestPaths( (donald)-[*]-(c) )
RETURN *
```

![](/static/images/trumpworld-us-contracting-data-neo4j/vornado-graph-trump-2.svg)
*ALL shortest paths from a specific contract to Donald Trump.*


If we expand the length of the path a bit we can see a few more potentially interesting connections:

```cypher
MATCH (c:Contract {piid: "DJU4500517056"})
MATCH (c)<-[:ISSUED_CONTRACT]-(a:Agency)
MATCH (donald:Person {name: "DONALD J. TRUMP"})
MATCH p=( (donald)-[*1..6]-(c) )
RETURN *
```


![](/static/images/trumpworld-us-contracting-data-neo4j/vornado-graph-trump-all.svg)
*ALL shortest paths from a specific contract to Donald Trump.*


### Cabinet Nominees

What connections do Trump's cabinet nominees have to contract vendors and parent companies of contract vendors?

```cypher
MATCH (donald:Person {name: "DONALD J. TRUMP"})<-[r:NOMINEE]-(cabinet:Person)
MATCH p=(cabinet)--(o:Organization)-[*1..2]-(c:Contract)--(a:Agency)
RETURN *
```

![](/static/images/trumpworld-us-contracting-data-neo4j/cabinet-graph-all.svg)
*Connections from Trump's cabinet nominees to government contracts.*

Let's take a closer look at the connection between some specific cabinet nominees to government contracts.

#### Rex Tillerson

Rex Tillerson is Trump's nominee for Secretary of State. He is also the former CEO and chairman of Exxon Mobil. Let's explore his connections in Trumpworld.


```cypher
MATCH (p:Person {name:"REX TILLERSON"})--(o:Organization {name: "EXXON MOBIL CORPORATION"}),
 path=(o)-[*1..2]-(c:Contract)--(a:Agency)
RETURN *
```

![](/static/images/trumpworld-us-contracting-data-neo4j/tillerson.png)
*Connections from Rex Tillerson to government contracts, through Exxon Mobil.*

We can see that Exxon Mobil itself was only awarded two contracts, one from the Department of the Army and another from the Department of the Navy. However, because we are modeling parent companies as well we can see that a subsidiary of Exxon Mobil (Mobil Oil Guam Inc) has been awarded significant contracts from the Department of the Navy and the Defense Logistics Agency.


#### James Mattis

James Mattis is Trump's nominee for Secretary of Defense. 

```cypher
MATCH (p:Person {name:"JAMES MATTIS"})--(o:Organization),
 path=(o)-[*1..2]-(c:Contract)--(a:Agency)
RETURN *
```


![](/static/images/trumpworld-us-contracting-data-neo4j/mattis.png)
*James Mattis has a direct connection to a company that is a vendor for several government contracts issued by departments that he oversees as Secretary of Defense.*

## Connections to Trump

Are there any direct connections from Trump to Organizations receiving government contracts?

```cypher
MATCH (p:Person) WHERE p.name CONTAINS "DONALD J. TRUMP"
MATCH (p)-[]-(o:Organization)-[:AWARDED_CONTRACT]->(c:Contract),
      (c)<-[:ISSUED_CONTRACT]-(a:Agency)
RETURN *
```

![](/static/images/trumpworld-us-contracting-data-neo4j/trump-palace.png)
*A government contract awarded to a company with a direct connection to Donald Trump.*

This is of course the contract that [allows the Secret Service to rent space in Trump Tower](http://www.politico.com/story/2016/12/secret-service-faces-massive-bill-for-protecting-trump-232153) for protecting Trump and his associates.

## What can you find?

All of the data and queries used here are public, so feel free to reproduce the data in your own instance of Neo4j. Neo4j is open source and free to use, you can get it [here](https://neo4j.com/download). To make working with this data even easier, we've created a [Neo4j Sandbox Trumpworld example](https://neo4j.com/sandbox-v2). In addition there are some embedded Neo4j Browser Guides for working with the dataset (introducing Cypher, interesting queries, applying social network analysis techniques to the dataset, etc. )  

**You can spin up your own Trumpworld Graph Neo4j instance by signing into the Sandbox [here](https://neo4j.com/sandbox-v2).**

[![](/static/images/trumpworld-us-contracting-data-neo4j/trump-guides.png)](https://neo4j.com/sandbox-v2)
*The Neo4j Sandbox instance includes Neo4j Browser Guides for working with the data in Neo4j*

Here are some ideas to get your started:

### Suggested areas for research

**What can we learn about cabinet appointees and their connections?**

```cypher
MATCH p=(a)-[r]-(b)
WHERE r.connection CONTAINS "Nominee"
RETURN p
```

**Extending the dataset. What other data sources could we include in the graph to enhance our understanding?**

* Campaign contributions from the FEC
* Nonprofit and foundation disclosure data
* Combine with related data from littlesis.org, "the involuntary registry of the 1%"
* Add people involved with the election campaign
* Add more detailed information on the organizations and their setup (location, leadership, deals), e.g. by querying APIs like OpenCorporates or by integrating with the data from Aleph the OCCRP data portal  

**Can we use centrality algorithms to find "influencers" in the network?**

For example, we can examine the connections of the people with the highest PageRank score in the network:


```cypher
// Find the first degree connections of the people with the highest PageRank score
MATCH (c:Person)
WITH collect(c) AS people
CALL apoc.algo.pageRank(people) YIELD node, score
WITH node, score ORDER BY score LIMIT 5
MATCH p=(node)-[]-()
RETURN p
```

**What are areas you're interested in exploring?**

By filtering on names and connection types we can explore the networks around key people and organizations. For example:


```cypher
// Find the second degree network of Goldman Sachs
MATCH (o:Organization) WHERE o.name CONTAINS "GOLDMAN SACHS"
MATCH p=(o)-[*2]-()
RETURN p
```


## Further work

Hopefully this post demonstrates what I think are the two most important benefits of using a graph database when doing data journalism type analysis:

1. *The intuitiveness of the data model and query language.* Thinking of your data in terms of graphs allows you to focus on connections in the data, which is often the interesting part when it comes to finding stories in data.
1. *The ability to extend the data model by combining datasets.* The flexibility of the property graph model allows for easily combining datasets and querying across them, as we did here with Trumpworld + USASpending.

There is still much work to be done with this data. Here are some ideas for further improvement:

* The vendor - organization matching was done using an exact string match. There are certainly companies that were not found due to spelling inconsistencies ("EXXONMOBIL" vs "EXXON MOBIL CORP" vs "EXXON MOBIL CORPORATION"). To improve the matching rate, we could resolve the Dun & Bradstreet DUNS number for each organization in Trumpworld (since USASpending data includes DUNS number). Or explore using fuzzy string matching to find high probability matches.
* Adding loan and grant information. In addition to grants, USASpending includes information on loan and grant recipients from the federal government.
* Other datasets. What other dataset would be interesting to add? Campaign finance, foundation / non-profit data?


One of the powers of working with graph databases is the ability to combine disparate datasets and query across them. Today we will look at how we can combine the BuzzFeed Trumpworld graph with data about federal government contracts from USASpending.gov, allowing us to examine any government contracts that were awarded to organizations that appear in Trumpworld.

Combining The BuzzFeed Trumpworld Graph with Government Contracting Data in Neo4j


Neo4j 3.0 introduced the concept of [user defined procedures](https://neo4j.com/docs/developer-manual/current/procedures/): code written in Java (or any JVM language) that is deployed to the database and callable from Cypher. User defined procedures are an alternative to unmanaged extensions, with the key difference that user defined procedures are callable from Cypher (instead of extending the http REST endpoints). This allows for an alternative API for [Neo4j's spatial extension](https://github.com/neo4j-contrib/spatial) - namely, exposing Neo4j Spatial's functionality through Cypher!

![](/static/images/neo4j-spatial-procedures-congressional-boundaries/spatial_query.png)

If you're not familar with [Neo4j Spatial](https://github.com/neo4j-contrib/spatial), it's an extension to Neo4j that implements spatial indexing that enables import, storage, and querying of spatial data in Neo4j. After the introduction of user defined procedures in Neo4j many procedures were added to spatial, making it much easier to use - instead of making separate http requests to the endpoints exposed by spatial or using the spatial Java API a user can now simply call spatial procedures directly from Cypher (and make use of the official language drivers and the efficient Bolt binary protocol).

Today I got around to updating [legis-graph-spatial](http://legis-graph.github.io/legis-graph-spatial/) to use these new spatial procedures, instead of the spatial REST API it was using previously. Here's a brief overview of the update:

* Simplified congressional district boundaries
* Update to Neo4j 3.04 and spatial 0.20
* Geospatial querying using spatial procedures from Cypher
* Update to official Neo4j Javascript driver using Bolt protocol

# Legis-graph-spatial

If you haven't seen it before, legis-graph-spatial provides a visual way to explore US Congress members topics of influence by district. The user clicks on a map to see who represents that area in the House, as well as information about the committees that representative serves on and over what issues they have influence in Congress. This is all powered by Neo4j. See [this post](http://www.lyonwj.com/2016/03/21/legis-graph-spatial-indexing/) for more info.

![](/static/images/neo4j-spatial-procedures-congressional-boundaries/ca_san_mateo.png)
*Legis-graph-spatial. Explore legislator topics of influence by district.*

# Simplified Congressional District Boundaries

Previously I was using a much higher resolution version of the congressional district boundaries then was necessary. I took this opportunity to replace the WKT boundaries for each district using this lower resolution data from [Govtrack.](https://gis.govtrack.us/map/demo/cd-2012/).

This Python script crawls the gis.govtrack.us API to fetch WKT format boundaries for each congressional district and saves it in a CSV file that we can later import into Neo4j easily as part of the [LOAD CSV import query for legis-graph](https://github.com/legis-graph/legis-graph/blob/master/quickstart/114/legis_graph_import_114.cypher):


```python

'''
Fetch simplified WKT boundaries for 2014 congressional districts and
save in CSV format:
    state,district,polygon

'''
import requests
import csv

BASE_URL = "https://gis.govtrack.us"
CD_2014_URL = "/boundaries/cd-2014/?limit=500"


# get meta boundary
r = requests.get(BASE_URL + CD_2014_URL)
j = r.json()

boundaries = j['objects']

with open('cb_2014_districts.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerow(['state', 'district', 'polygon'])

    for b in boundaries:
        p = str.split(b['name'], '-')
        r = requests.get(BASE_URL + b['url'] + 'simple_shape?format=wkt')
        wkt = r.text
        writer.writerow([p[0], p[1], wkt])

```

# Upgrading And Installing Neo4j Spatial

The server for legis-graph-spatial is running on a Digital Ocean vps instance, so I upgraded to Neo4j 3.0.4 using the Neo4j [Debian package](http://debian.neo4j.org/) which will install Neo4j as a service. Once I installed Neo4j I built and installed spatial:

```shell

git clone https://github.com/neo4j-contrib/spatial.git
cd spatial
mvn clean install -DskipTests
cp target/neo4j-spatial-0.20-neo4j-3.0.3-server-plugin.jar $NEO4J_HOME/plugins/.
$NEO4J_HOME/bin/neo4j restart

```


Instead of building spatial you can also download the spatial plugin jar file from Github:

```shell
wget https://github.com/neo4j-contrib/m2/blob/master/releases/org/neo4j/neo4j-spatial/0.20-neo4j-3.0.3/neo4j-spatial-0.19-neo4j-3.0.3-server-plugin.jar?raw=true
cp neo4j-spatial-0.20-neo4j-3.0.3-server-plugin.jar $NEO4J_HOME/plugins/.
$NEO4J_HOME/bin/neo4j restart
```

Then I used the [legis-graph import script](https://github.com/legis-graph/legis-graph/blob/master/quickstart/114/legis_graph_import_114.cypher) with the [LazyWebCypher tool for easily running multi statement Cypher scripts](http://www.lyonwj.com/LazyWebCypher/?file=https://raw.githubusercontent.com/legis-graph/legis-graph/master/quickstart/114/legis_graph_import_114.cypher) from the browser to load legis-graph data for the 114th Congress.

# Indexing Districts

Once the data is loaded we want to add the `District` nodes to a spatial index so we can do things like find the Congressional District that contains a certain latitude and longitude.

We'll use a spatial procedure, `spatial.addWKTLayer` to create a WKT layer:

```cypher
CALL spatial.addWKTLayer('geom', 'wkt')
```

Then we'll match on all `District` nodes and add them to the WKT layer we just created (again using a spatial procedure, in this case `spatial.addNodes`):

```cypher
MATCH (d:District)
WITH collect(d) AS districts
CALL spatial.addNodes('geom', districts) YIELD node
RETURN count(*)
```

# Querying

Now that the `District` nodes have been added to the spatial layer, we can perform an indexed geospatial query to find the Congressional district given a latitude and longitude. And since these `District` nodes are part of [legis-graph](https://github.com/legis-graph/legis-graph), it's just a simple graph traversal to find the Legislator that represents that District and their topics of influence:


```cypher
CALL spatial.withinDistance('geom',
  {latitude: 37.563440, longitude: -122.322265}, 1) YIELD node AS d
WITH d, d.wkt AS wkt, d.state AS state, d.district AS district LIMIT 1
MATCH (d)<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:SERVES_ON]->(c:Committee)
MATCH (c)<-[:REFERRED_TO]-(b:Bill)
MATCH (b)-[:DEALS_WITH]->(s:Subject)
WITH wkt, state, district, l.govtrackID AS govtrackID, l.lastName AS lastName,
  l.firstName AS firstName, l.currentParty AS party, s.title AS subject,
  count(*) AS strength, collect(DISTINCT c.name) AS committees
ORDER BY strength DESC LIMIT 10
WITH wkt, state, district, {lastName: lastName, firstName: firstName,
  govtrackID: govtrackID, party: party, committees: committees} AS legislator,
  collect({subject: subject, strength: strength}) AS subjects
RETURN {legislator: legislator, subjects: subjects, state: state,
  district: district} AS r, wkt LIMIT 1
```

![](/static/images/neo4j-spatial-procedures-congressional-boundaries/spatial_query_table.png)
*Querying legis-graph using indexed geospatial query from Cypher*

# Using the official Neo4j Javascript Driver

Previously I was making an http request to the spatial endpoint to query the district and another to the Cypher transactional endpoint to query for topics of influence, but now we can combine this all into one query as seen above.

Also, we were using a jQuery AJAX POST request to the transaction Cypher HTTP endpoint to execute our Cypher query:

```js
 /**
     *  Run a Cypher query, call the callback with results
     */
    function makeCypherRequest(statements, callback) {

        var url = baseURI.replace(/\/db\/data.*/,"") + "/db/data/transaction/commit";

        $.ajax({
            type: 'POST',
            data: JSON.stringify({
                statements: statements
            }),
            contentType: 'application/json',
            url: url,
            error: function(xhr, statusText, errorThrown){
                callback("Error", null);
            },
            //headers: authHeader(),
            success: function(data) {
                console.log(data);
                callback(null, data);

            }
        });
    }
```



 With the release of Neo4j 3.0 we can instead make use of the new Bolt binary protocol to send less data over the wire and save time on serialization. For this we'll send our query using the [neo4j-javascript-driver](https://github.com/neo4j/neo4j-javascript-driver).

The neo4j-javascript-driver has a version for node.js based applications and a version for the browser. We can include the browser version in our simple app like this:

```html
<script src="/lib/neo4j-web.min.js"></script>
```

Then to execute the Cypher query we saw above that does our spatial lookup and parse the data:

```js
var driver = neo4j.v1.driver(neo4jURI);
var session = driver.session();

var districtParams = {
    "layer": "geom",
    "lon": latlng.lng,
    "lat": latlng.lat,
    "distanceInKm": distance
}

session
    .run(subjectsQuery, districtParams)
    .then(function(result){
        result.records.forEach(function(record) {
            console.log(record);

            var points = parseWKTPolygon(record.get("wkt"));
            var districtInfo = record.get("r");
            districtInfo["points"] = points;

            addDistrictToMap(districtInfo, latlng);

        });
});
```

As you can see we are using a promise for dealing with the asynchronous behavior of a database request. Alternatively we could act on the data as a stream, by subscribing to the data event:

```js
session
  .run(subjectsQuery, districtParams)
  .subscribe({
    onNext: function(record) {
     console.log(record);

     var points = parseWKTPolygon(record.get("wkt"));
     var districtInfo = record.get("r");
     districtInfo["points"] = points;
     addDistrictToMap(districtInfo, latlng);
    }
  });
```

However since we only expect to receive one row back we don't need to process as a stream.

You can give the new legis-graph-spatial version a try [here](http://legis-graph.github.io/legis-graph-spatial/) and as always the full code is available on [Github.](http://github.com/legis-graph/legis-graph-spatial)

Updating legis-graph-spatial to make use of the new spatial procedures in Neo4j 3.0 and the official Neo4j Javascript driver. Procedures provide a new API for interacting with the Neo4j spatial extension and are callable from Cypher.

Using Neo4j Spatial Procedures in legis-graph-spatial

Using graphs to analyze character interactions in the popular Game of Thrones series. Social network analysis with Neo4j.

Analyzing the Graph of Thrones



![](/static/images/legis-graph-spatial-indexing/lg_denver.png)
**Figure 1: legis-graph-spatial: An interactive map visualization using Neo4j and Mapbox.**

In this post we will show how we can use geospatial indexing with Neo4j. We will extend the [legis-graph](https://github.com/legis-graph/legis-graph) dataset (a graph of US Congress), adding US Congressional districts to the graph. We will use [neo4j-spatial](https://github.com/neo4j-contrib/spatial), a plugin for Neo4j, to index the geometries of the Congressional districts. Finally, we will show how we can connect Neo4j to Mapbox, a mapping framework, to make an interactive geographical visualization in the browser. Let's get started!


## Overview of legis-graph

![](/static/images/legis-graph-spatial-indexing/lgdatamodel.png)

**Figure 2: The legis-graph data model. This dataset describes the activities of US Congress. The data comes from [govtrack.us](http://govtrack.us).**

[Legis-graph](https://github.com/legis-graph/legis-graph) is a dataset that models US Congress as a graph. After all US Congress is a collaboration network and should be modeled as such. By modeling the data as a graph in Neo4j we can write powerful graph traversal queries to answer questions like ["What are the topics of bills certain legislators are sponsoring?"](http://www.lyonwj.com/2015/09/20/legis-graph-congressional-data-using-neo4j/) and ["Who is the most influential Senator in Congress with influence over certain topics?"](http://www.lyonwj.com/2015/10/11/congressional-pagerank/).

![](/static/images/legis-graph-spatial-indexing/lg_body_party_state.png)

**Figure 3: An example of a query of the legis-graph dataset. Legislators who represent California by party and congressional body.**

If you'd like to experiment with legis-graph in Neo4j there is a read only Neo4j instance online available [here.](http://bit.ly/1QhTh0s)

## An interactive map visualization using Neo4j and Mapbox

There are many dimensions to explore in this dataset, but I wanted to do more with the spatial component. We have state nodes in the graph, but I wanted some way to interactively explore the data using a map. As I was preparing my presentation for the [NICAR data journalism conference](http://www.slideshare.net/lyonwj/finding-insights-in-connected-data-using-graph-databases-in-journalism) this year, I took the opportunity to extend legis-graph and build a simple interactive map interface that enables querying legis-graph based on spatial geometries (Congressional districts) and visualizing the results as part of the map.

 You can try the initial version [here](http://legis-graph.github.io/legis-graph-spatial/) - just click anywhere in the US to get started. I have lots of plans for adding additional functionality, so stay tuned for updates!

## Overview of neo4j-spatial

The first component we need for this project is some way to index geometries in the database and a method for querying those geometries. Enter neo4j-spatial! [Neo4j-spatial](https://github.com/neo4j-contrib/spatial) is a Neo4j plug-in that facilitates geospatial operations on data stored in Neo4j. Among other operations, it enables indexed geospatial queries by implementing an [R-tree](https://en.wikipedia.org/wiki/R-tree), in this case in the graph because an R-tree is itself a graph.

There are several [great](http://maxdemarzi.com/2014/01/31/neo4j-spatial-part-1/) [examples](http://maxdemarzi.com/2014/02/11/neo4j-spatial-part-2/) [and](http://www.lyonwj.com/using-neo4j-spatial-and-leaflet-js-with-mapbox) [blog](http://www.lyonwj.com/mapping-the-worlds-airports-with-neo4j-spatial-and-openflights-part-1) [posts](https://structr.org/blog/distance-queries-with-neo4j-spatial) [out there](http://spatialcypherdemo.herokuapp.com/) that show how to index and query point (X,Y) geometries, like points of interest, but here we are dealing with more complex polygon geometries (Congressional districts). No worries, neo4j-spatial supports these more complex geometries using the [WKT, or Well Known Text format](https://en.wikipedia.org/wiki/Well-known_text)!

## Adding Congressional districts to legis-graph

Neo4j-spatial allows us to index nodes that contain vector geometries specified as properties on the nodes. In this case we need to extend the legis-graph data model to include `District` nodes and store their geometry as a string property in WKT format.

The geometries of each congressional district are available [here](https://www.census.gov/geo/maps-data/data/cbf/cbf_cds.html) as shapefiles. Once we've downloaded the files we need to read the geometries of each district and convert to WKT format. There are many tools available that can do this but I used the [pyshp Python library](https://github.com/GeospatialPython/pyshp) to do this and wrote them to a CSV file to be used during the import process for legis-graph.


## The legis-graph datamodel, with districts

One of the benefits of the graph data model is the ability to transform the data model easily as our requirements change. Previously, we had a direct `REPRESENTS` relationship from the `Legislator` node to the `State` node. Now that we have more fine grained data (Congressional districts) we can easily extend the data model by creating `District` nodes and updating our relationships in the graph:

![](/static/images/legis-graph-spatial-indexing/district_datamodel_1.svg)
**Figure 4: Adding District nodes to the graph. The geometry of each district is stored as a string property on the node in WKT format.**


## Creating and adding nodes to the spatial layer

Now that we've stored the Congressional geometries in the graph we need to tell neo4j-spatial to add these nodes to the spatial index. We can do this by making a series of HTTP REST requests to the Neo4j server (once we've installed neo4j-spatial).


### Create the layer

First, we'll make an HTTP request to create a WKT layer:

```md

POST http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addEditableLayer

Accept: application/json; charset=UTF-8

Content-Type: application/json

{
  "layer" : "geom",
  "format" : "WKT",
  "nodePropertyName" : "wkt"
}
```
**From the [neo4j-spatial documentation](http://neo4j-contrib.github.io/spatial/#rest-api-create-a-wkt-layer), create a WKT layer.**

### Add nodes to the layer

Now that we've created it, we need to add our `District` nodes to the spatial layer. We can do this using another HTTP request:

```md
Example request

POST http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addNodeToLayer

Accept: application/json; charset=UTF-8

Content-Type: application/json

{
  "layer" : "geom",
  "node" : "http://localhost:7575/db/data/node/54"
}

```
**From the [neo4j-spatial documentation](http://neo4j-contrib.github.io/spatial/#rest-api-add-a-node-to-the-spatial-index), add a node to the layer.**


As I mentioned earlier, neo4j-spatial uses an in-graph RTree index. We can see how the graph updates once we start adding nodes to the layer:

![](/static/images/legis-graph-spatial-indexing/district_datamodel_2.svg)
**Figure 5: Data model after creating the in-graph index.**

I wrote a simple Python script to iterate over the `District` nodes that had not yet been indexed and add them to the spatial layer:

```python

import requests
from py2neo import Graph

# A Neo4j instance with Legis-Graph
graph = Graph("http://52.70.212.93/db/data")
baseURI = "http://52.70.212.93"

# this function will add a node to a spatial layer
def addNodeToLayer(layer, nodeId):
    addNodeToLayerParams = {"node": baseURI+ "/db/data/node/" + str(nodeId), "layer": layer}
    r = requests.post(baseURI + "/db/data/ext/SpatialPlugin/graphdb/addNodeToLayer", json=addNodeToLayerParams)

# Find District nodes that have wkt property and are not part of the spatial index.
# Add these nodes to the layer
getIdsQuery = "MATCH (n:District) WHERE has(n.wkt) AND NOT (n)-[:RTREE_REFERENCE]-() RETURN id(n) AS n"
results = graph.cypher.execute(getIdsQuery)
for record in results:
    nodeId = record.n
    addNodeToLayer("geom", nodeId)
```

**This Python snippet queries the graph for nodes that have not yet been added to the spatial index and makes a REST request to add them to the index.**

## Querying for closest district

Now that we have our geospatial data indexed in the graph we need some way to query it! Again we'll use an HTTP request to the REST API that neo4j-spatial exposes:


```md
Example request

POST http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance

Accept: application/json; charset=UTF-8

Content-Type: application/json

{
  "layer" : "geom",
  "pointX" : 15.0,
  "pointY" : 60.0,
  "distanceInKm" : 100
}
```
**From the [neo4j-spatial documentation](http://neo4j-contrib.github.io/spatial/#rest-api-find-geometries-within--distance), finding geometries within distance of a point**

## Integrating with Mapbox

Now it's time to bring in our map. We want to allow our user to click somewhere on the map, find the closest Congressional district and query legis-graph for information about that legislator, including the Committees on which they serve and the topics of bills referred to those committees. This will give us some idea of the topics over which that legislator has influence.

We'll use the awesome [Mapbox JS API](https://www.mapbox.com/developers/) for our map. Mapbox allows us to embed an interactive map in our web page and provides functions for interacting with the map. In our case the specific features we're interested in are responding to clicks on the map and annotating the map by drawing polygons and popups.

The first step, after including the Mapbox library is to define the map object and specify a click handler function that is called when a user clicks the map:

```js
L.mapbox.accessToken = MB_API_TOKEN;
var map = L.mapbox.map('map', 'mapbox.streets')
  .setView([39.8282, -98.5795], 5);

map.on('click', function(e) {
  clearMap(map);
  getClosestDistrict(e);
});
```
**Create the map and define a click handler for the map.**

Here `getClosestDistrict` is a function that will take the latitude and longitude of the mouse click and make a request to the `/findGeometriesWithinDistance` endpoint exposed by Neo4j spatial plugin to find the closest Congressional district, then executes a Cypher query to find the committees, bills and bill subjects that legislator has overseen in their committees.

```js

/**
  *  Find the District for a given latlng.
  *  Find the representative, commitees and subjects for that rep.
  */
function infoDistrictWithinDistance(latlng, distance) {

  var districtParams = {
    "layer": "geom",
    "pointX": latlng.lng,
    "pointY": latlng.lat,
    "distanceInKm": distance
  };

 var districtURL = baseURI + findGeometriesPath;
 makePOSTRequest(districtURL, districtParams, function (error, data) {

   if (error) {
    console.log("Error");
   } else {
    console.log(data);

   var params = {
    "state": data[0]["data"]["state"],
    "district": data[0]["data"]["district"]
   };

   var points = parseWKTPolygon(data[0]["data"]["wkt"]);

   makeCypherRequest([{"statement": subjectsQuery, "parameters": params}], function (error, data) {

    if (error) {
      console.log("Error");
    } else {
      console.log(data);

      var districtInfo = data["results"][0]["data"][0]["row"][0];
      districtInfo["points"] = points;
      districtInfo["state"] = params["state"];
      districtInfo["district"] = params["district"];
      console.log(districtInfo);

      addDistrictToMap(districtInfo, latlng);
    }
   });
 }
});
```

We also return the WKT string that defines the district geometry so that we can annotate the map with a polygon for the district. However, we need to parse that from a WKT string into an array of points:

```js
/**
 *  Converts Polygon WKT string to an array of [x,y] points
 */
function parseWKTPolygon(wkt) {
  var pointArr = [];
  var points = wkt.slice(10, -3).split(",");

  $.each(points, function(i,v) {
    var point = $.trim(v).split(" ");
    var xy = [Number(point[1]), Number(point[0])];
    pointArr.push(xy)
  });

  return pointArr;
}
```


Another useful utility function that we define uses the jQuery ajax function and the Neo4j transactional Cypher HTTP endpoint to execute a parameterized Cypher statement:


```js
/**
*  Run a Cypher query
*/
function makeCypherRequest(statements, callback) {

  var url = baseURI + "/db/data/transaction/commit";

  $.ajax({
    type: 'POST',
    data: JSON.stringify({
      statements: statements
    }),
    contentType: 'application/json',
    url: url,
    error: function(xhr, statusText, errorThrown){
      callback("Error", null);
    },
    //headers: authHeader(), // add basic headers if necessary
    success: function(data) {
      console.log(data);
      callback(null, data);
    }
  });
}
```

Once we've gotten the data back from Neo4j and parsed the WKT into an array of points we can annotate the map:

## Annotating the map

```js

/**
 *  Add District polygon and legislator popup to map
 *
 * @param data
 * @param latlng
 */
function addDistrictToMap(data, latlng) {
  polygon_points = data["points"];

  popuptext = buildPopup(data);

  var polyline = L.polygon(polygon_points, {color: 'brown'}).addTo(map);
  map.fitBounds(polyline.getBounds());

  var popup = L.popup({keepInView: true, minWidth: 350, maxWidth: 1000});
  popup.setLatLng(latlng)
    .setContent(popuptext)
    .openOn(map);
    popup.update();
}
```

Once we have it running it looks like this:

![](/static/images/legis-graph-spatial-indexing/legis-graph-spatial.gif)
**Legis-graph-spatial demo. Try it [here](http://legis-graph.github.io/legis-graph-spatial/).**


There are a few portions of the code that I didn't go over, such as building the map popup, but all code is available in [this GitHub repository](https://github.com/legis-graph/legis-graph-spatial).


Adding geospatial indexing and map visualizations to legis-graph

Adding Geospatial Indexing to legis-graph With Neo4j and Mapbox


import YouTube from '../../components/YouTube';

<YouTube youTubeId="BVMx24dtko0" />

This week I'm preparing a presentation for [DataDay Texas](http://datadaytexas.com/) about [Natural Language Processing with graph databases and Neo4j](http://datadaytexas.com/sessions). While doing some research I came across a great quote from [Matt Biddulph](https://www.hackdiary.com/):

>“Nearly all text processing starts by transforming text into vectors” - Matt Biddulph

He is referring of course to building a vector space model from a text corpus as the first step of text processing. In the context of NLP with graph databases, the first step is building a graph model from the text corpus. In many cases this is a simple word adjacency graph.

Another resource I stumbled across on the internet this week is the Best Buy API. This is a great resource for playing around with text processing. Specifically, the Reviews endpoint provides access to a rich series of user generated text that we can use for text processing. I'm specifically interested in text summarization and opinion mining so this is a perfect data set.

![Best Buy Developer](/static/images/nlp-graph-model/bestbuy-logo.png)

After [registering with the Best Buy API](https://developer.bestbuy.com/documentation/getting-started) and retrieving an API token this simple Python script can be used to fetch the first 500 product reviews for any Best Buy product and insert into Neo4j following the model of a word adjacency graph. Just replace the `API_KEY` and `SKU` variables to retrieve product reviews for any product.

```python
from py2neo import Graph
import json
import requests

API_KEY = "YOUR_API_KEY_HERE"
SKU = "9439005" # Kindle Paperwhite
REQUEST_URL = "https://api.bestbuy.com/v1/reviews(sku={sku})?apiKey={API_KEY}&show=comment,id,rating,reviewer.name,sku,submissionTime,title&pageSize=100&page={page}&sort=comment.asc&format=json"

INSERT_QUERY = '''
WITH split(tolower({comment}), " ") AS text
UNWIND range(0,size(text)-2) AS i
MERGE (w1:Word {name: text[i]})
MERGE (w2:Word {name: text[i+1]})
CREATE UNIQUE (w1)-[:NEXT]->(w2)
'''

graph = Graph()

for i in range(1,6):  # fetch first 500 reviews (5 pages at 100 per page)
    r = requests.get(REQUEST_URL.format(sku=SKU, API_KEY=API_KEY, page=str(i)))
    data = r.json()
    for comment in data["reviews"]:
        graph.cypher.execute(INSERT_QUERY, parameters={'comment': comment["comment"]})
```

Once the script runs, a snippet of the graph looks like this:

![](/static/images/nlp-graph-model/kindle_reviews.png)

And now we can use Cypher to start analyzing our text corpus. For more reading on this area be sure to check out these great posts from [Max](http://maxdemarzi.com/2012/08/10/summarize-opinions-with-a-graph-part-1/) and [Michael](http://jexp.de/blog/2015/01/natural-language-analytics-made-simple-and-visual-with-neo4j/).


Finding real world text datasets to analyze can make playing around with NLP tasks much more interesting. This quick post shows how to build a word adjacency graph in Neo4j using product reviews from the Best Buy API.

Building a text adjacency graph from product reviews with the Best Buy API



A common task when working with data from APIs is crawling the API and inserting the results in a database. In this example we'll import data into Neo4j by crawling a public API, building a graph in Neo4j as we go along. We'll use Neo4j as a queuing mechanism to store URLs for placeholder resources waiting to be fetched.

Specifically, we'll use [SWAPI.co](http://swapi.co) to build a graph of information from the Star Wars universe. SWAPI contains data about characters, planets, starships (and much more!) that appear in the Star Wars universe. Much of the data is collected from [Wookieepedia](http://starwars.wikia.com/wiki/Main_Page).

## The Data Model

![](/static/images/crawling-public-api-neo4j/swapi.png)

The graph data model we'll use when building the graph.

This example will use Python and py2neo (a Python driver for Neo4j) to fetch data from [SWAPI.co](http://swapi.co) and is available in an [iPython Notebook](https://github.com/johnymontana/SWAPI-graph/blob/master/SWAPI.ipynb).

To see the full example please check out the iPython Notebook [here](http://nbviewer.ipython.org/github/johnymontana/SWAPI-graph/blob/master/SWAPI.ipynb).


A common task when working with data from APIs is crawling the API and inserting the results in a database. In this example we will import data into Neo4j by crawling a public API, building a graph in Neo4j as we go along. We will use Neo4j as a queuing mechanism to store URLs for placeholder resources waiting to be fetched.

Crawling APIs Using Neo4j - Building The Star Wars Graph



As we saw [previously](http://www.lyonwj.com/2015/09/20/legis-graph-congressional-data-using-neo4j/), [legis-graph](https://github.com/legis-graph/legis-graph) is an open source software project that imports US Congressional data from [Govtrack](http://govtrack.us) into the Neo4j graph database. This post shows how we can apply graph analytics to US Congressional data to find influential legislators in Congress. Using the Mazerunner open source graph analytics project we are able to use Apache Spark GraphX alongside Neo4j to run the PageRank algorithm on a collaboration graph of US Congress.

While Neo4j is a powerful graph database that allows for efficient [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) queries and graph traversals using the Cypher query language, it is not optimized for global graph algorithms, such as PageRank. [Apache Spark](http://spark.apache.org/) is a distributed in-memory large-scale data processing engine with a graph processing framework called [GraphX](http://spark.apache.org/graphx/). GraphX with Apache Spark is very efficient at performing global graph operations, like the PageRank algorithm. By using Spark alongside Neo4j we can enhance our analysis of US Congress using legis-graph.

## Neo4j + Apache Spark: Overview of the Mazeruner project

[Mazerunner](https://github.com/kbastani/neo4j-mazerunner) is a powerful open source graph analytics project that enables us to run Apache Spark GraphX jobs on a subgraph exported from Neo4j. Mazerunner makes use of Docker to allow for easy deployment. From the Mazerunner [GitHub README](https://github.com/kbastani/neo4j-mazerunner#graph-analytics-for-neo4j):

> This docker image adds high-performance graph analytics to a Neo4j graph database. This image deploys a container with Apache Spark and uses GraphX to perform ETL graph analysis on subgraphs exported from Neo4j. The results of the analysis are applied back to the data in the Neo4j database.

Mazerunner was created by [Kenny Bastani](https://twitter.com/kennybastani) who has written some [great posts](http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html) [about using](http://www.kennybastani.com/2015/03/spark-neo4j-tutorial-docker.html) [Mazerunner for](http://www.kennybastani.com/2014/11/graph-analytics-docker-spark-neo4j.html) [graph analytics with Neo4j](https://www.airpair.com/apache-spark/posts/distributed-pagerank-using-apache-spark-and-neo4j).

## Using docker compose

We'll use [Docker](https://www.docker.com/) to run Mazerunner so be sure to [install Docker](https://docs.docker.com/installation/) before moving on. Docker will allow us to easily deploy Hadoop HDFS, Neo4j and Apache Spark.  We'll use [docker compose](https://docs.docker.com/compose/) to manage our containers and define the port bindings. To start, we'll create a yaml file that specifies the containers we'd like to spin up, how they should be linked to each other and which ports should be exposed:


```yaml
hdfs:
 image: sequenceiq/hadoop-docker:2.4.1
 command: /etc/bootstrap.sh -d -bash
mazerunner:
 image: kbastani/neo4j-graph-analytics:latest
 links:
  - hdfs
graphdb:
 image: kbastani/docker-neo4j:latest
 ports:
  - "7474:7474"
  - "1337:1337"
 volumes:
  - /opt/data
 links:
  - mazerunner
  - hdfs
```

> See [this GitHub page](https://github.com/kbastani/spark-neo4j) for more information about running Mazerunner with Docker compose.

The yaml file defines the three containers we'll be using (Hadoop HDFS, Mazerunner service, and Neo4j), how they should be linked and what ports should be exposed so that we can access Neo4j. We can start this configuration with a single command:

```bash
docker-compose up
```

![](/static/images/congressional-pagerank/docker-compose.gif)

We now have Hadoop HDFS, an empty Neo4j instance and Mazerunner service running in separate containers, but linked together (with some docker magic...). We can connect to Neo4j easily since the Neo4j ports are exposed. Since I'm using docker on a Mac, there is actually a docker virtual machine running locally. To connect to Neo4j I can use the IP address of this VM. The default docker VM IP address is 192.168.99.100 but we can confirm with the `docker-machine inspect` command:

```shell

lyonwjs-MacBook-Pro-2:~ lyonwj$ docker-machine inspect default | grep IPAddress
        "IPAddress": "192.168.99.100",
```

We can now connect to the Neo4j Browser at `http://192.168.99.100:7474/browser/` to verify that Neo4j is running and there is no data in the database.

Now we need some data...

## Importing legis-graph

If you haven't read my [previous post](http://www.lyonwj.com/2015/09/20/legis-graph-congressional-data-using-neo4j/) about the [legis-graph project](https://github.com/legis-graph/legis-graph) you should at least skim it over to see the data model we are working with and some of the queries that are possible.

Now that we have Neo4j running we can import legis-graph into the database, following the steps described [here](https://github.com/legis-graph/legis-graph#load-data). The steps involve fetching [GovTrack](https://www.govtrack.us/) data with rsync, using Python scripts to parse the JSON, YAML and CSV files provided by GovTrack into a series of CSV files that we can then import directly into Neo4j using Cypher scripts. The screencast below shows the exact commands so you can follow along. It's worth noting that we're using `neo4j-shell` to run the Cypher import scripts instead of the Neo4j Browser. To access the `neo4j-shell` in our Docker container we can first get a bash prompt with the command

```shell
docker exec -it CONTAINER_ID_HERE bash
```

where `CONTAINER_ID_HERE` is the id of the container running Neo4j, which we can find with the `docker ps` command. See the first part of the gif below. Once we have access to the shell in the container the steps to import legis-graph are identical to the those described in the [project readme](https://github.com/legis-graph/legis-graph#load-data)).

![](/static/images/congressional-pagerank/legis-graph-import.gif)

## Adding `INFLUENCED_BY` relationship

Once the import scripts have finished running we have data describing every US Congress Legislator, Bill and Committee for a given Congressional session (the results below are based on data from the 114th Congressional session, the most current session) in a Neo4j property graph. Our data model looks like this:

 ![](/static/images/congressional-pagerank/lgdatamodel.png)

In the data model we can see a relationship between our Legislator nodes and Bill nodes called `SPONSORED_BY`. From our memory of [Schoolhouse Rock](https://www.youtube.com/watch?v=Otbml6WIQPo) we know that every Bill has an initial Sponsor who introduces the Bill and several Cosponsors who support the Bill as well. In a political sense we can think of the Sponsor as influencing the Cosponsors to support the Bill. Although it's not explicitly modeled in our graph we can use this measure of influence to identify influential Legislators.

We will start by finding these implicit relationships in the Congressional [collaboration graph](https://en.wikipedia.org/wiki/Collaboration_graph) and creating explicit `INFLUENCED_BY` relationships. Anytime that a Bill was introduced by a Sponsor and then supported by Cosponsors, we will model that influence in our graph by creating an `INFLUENCED_BY` relationship from each Cosponsor to the Sponsor. For example, let's consider a specific Bill, HR5393, which was introduced and sponsored in the House by Representative Sam Graves. This Bill was then cosponsored by Representatives Mick Mulvaney, Steve Chabot, Marsha Blackburn, and Scott Tipton. We can observe that Rep Sam Graves demonstrated political leadership and influence over the cosponsors of this Bill by garnering its support, so we create an `INFLUENCED_BY` relationship from each cosponsor to Sam Graves in the graph to represent this political influence.


![](/static/images/congressional-pagerank/influenced_by_add.gif)

This Cypher query will update our graph and create these new `INFLUENCED_BY` relationships for all Bills:

```cypher
MATCH (b:Bill)-[r:SPONSORED_BY]->(sponsor:Legislator)
WHERE r.cosponsor = "0"
MATCH (b)-[s:SPONSORED_BY]->(cosponsor:Legislator)
WHERE s.cosponsor = "1"
CREATE (cosponsor)-[:INFLUENCED_BY]->(sponsor)
```


## Running PageRank on Legislators

We can use these new `INFLUENCED_BY` relationships as inputs for the PageRank algorithm to find influential legislators. [PageRank](https://en.wikipedia.org/wiki/PageRank) is an algorithm initially used by Google to measure the importance of webpages for Google Search. The same algorithm can be applied to many different types of networks to find significant nodes in the network. In our case we will be using PageRank to measure the political influence of legislators based on Bill sponsorship and cosponsorship.

>Note that this type of analysis is similar to the GovTrack [leadership analysis](https://www.govtrack.us/about/analysis#leadership) as explained in [this presentation](https://razor.occams.info/pubdocs/2012-10-08_LVI_slides.pdf). Both are based on the PageRank algorithm and the results are roughly similar.

## Running PageRank with Mazerunner

When we started our Docker containers earlier, one of those containers contained Mazerunner, a powerful tool for running Apache Spark GraphX jobs on HDFS with data from Neo4j. We can start a job in Mazerunner by making a simple `GET` request to Neo4j. The `GET` request only needs to specify the type of analysis we want to perform (in this case PageRank), and the relationship type on which we want to perform the analysis (`INFLUENCED_BY` in this case). Mazerunner will then take care of exporting the relevant subgraph to Spark, running the Spark GraphX job and then writing the results back to Neo4j. We can trigger this analysis from within the Neo4j browser with this command:

```md
:GET /service/mazerunner/analysis/pagerank/INFLUENCED_BY
```

![](/static/images/congressional-pagerank/pagerank2.gif)

Once the PageRank analysis job is complete, each Legislator node will be updated with a new `pagerank` property that contains the value of that Legislator's PageRank score based on the `INFLUENCED_BY` relationships. We can use this score as a proxy for political influence in Congress. Since this PageRank score was written back to Neo4j we can use Cypher queries to now take into account a Legislator's political influence when querying the graph. In the next section we'll answer some interesting questions about finding influencers in Congress using Cypher and our new Congressional PageRank score.

## Finding influential legislators

### Most influential Senators

```cypher
MATCH (l:Legislator) WHERE has(l.pagerank) WITH l, l.pagerank as pg
MATCH (l)-[:ELECTED_TO]->(b:Body {type: "Senate"})
RETURN l.firstName + l.lastName as name, pg ORDER BY pg DESC LIMIT 10;
```

```md
+-----------------------------------------+
| name               | pg                 |
+-----------------------------------------+
| "Orrin Hatch"      | 2.9756989829857203 |
| "Charles Grassley" | 2.6139548199234857 |
| "Benjamin Cardin"  | 2.1132989694887456 |
| "John Thune"       | 2.0930332521393904 |
| "Mike Lee"         | 2.0406616669993336 |
| "Lamar Alexander"  | 2.031176645832514  |
| "Jerry Moran"      | 1.9622461537176095 |
| "John Cornyn"      | 1.956690813571645  |
| "John Barrasso"    | 1.924180343470944  |
| "Jeff Flake"       | 1.8889703078024018 |
+-----------------------------------------+
10 rows
121 ms
```

### Most influential in House of Representatives

```cypher
MATCH (l:Legislator) WHERE has(l.pagerank) WITH l, l.pagerank as pg
MATCH (l)-[:ELECTED_TO]->(b:Body {type: "House"})
RETURN l.firstName + " " + l.lastName as name, pg ORDER BY pg DESC LIMIT 10;
```

```md
+------------------------------------------+
| name                | pg                 |
+------------------------------------------+
| "Diane Black"       | 8.58655429827745   |
| "Brett Guthrie"     | 6.062435816533887  |
| "Bob Goodlatte"     | 5.77444037360594   |
| "Erik Paulsen"      | 5.769744558672802  |
| "Sam Johnson"       | 5.51484568055709   |
| "Charles Boustany"  | 5.225312360342051  |
| "Carolyn Maloney"   | 4.879633494250425  |
| "Christopher Smith" | 4.799080848575352  |
| "Kevin Brady"       | 4.5629395513518265 |
| "Paul Gosar"        | 4.005903699546033  |
+------------------------------------------+
10 rows
63 ms
```

### Most influential Democrats in the House of Representatives

```cypher
MATCH (l:Legislator) WHERE has(l.pagerank) WITH l, l.pagerank as pg
MATCH (l)-[:ELECTED_TO]->(b:Body {type: "House"})
MATCH (l)-[:IS_MEMBER_OF]->(p:Party {name: "Democrat"})
RETURN l.firstName + " " + l.lastName as name, pg ORDER BY pg DESC LIMIT 10;
```

```md
+-------------------------------------------+
| name                 | pg                 |
+-------------------------------------------+
| "Carolyn Maloney"    | 4.879633494250425  |
| "Rosa DeLauro"       | 3.9910225707498417 |
| "Joe Courtney"       | 3.1157408773054067 |
| "Matthew Cartwright" | 3.082758638951927  |
| "Terri Sewell"       | 2.460045410055251  |
| "Jackie Speier"      | 2.184626502764663  |
| "Maxine Waters"      | 2.074914490498521  |
| "David Cicilline"    | 1.927110056774834  |
| "Henry Johnson"      | 1.8882492772785748 |
| "Robert Scott"       | 1.7769840317110424 |
+-------------------------------------------+
10 rows
62 ms
```


### Most influential for specific subjects

Here is where things get interesting and we start *really* leveraging the power of the graph. Simply by traversing the graph, we can find Legislators who have significant influence in Congress over specific subjects given the Commitees on which they serve and the types of Bills referred to those Committees.

#### Hong Kong

Let's take Hong Kong as a random example. Who are the Legislators with the highest level of influence (as measured by our Congressional PageRank score) who have influence over the subject of "Hong Kong", based on the Commitees on which they serve?

```cypher
MATCH (l:Legislator) WHERE has(l.pagerank) WITH l, l.pagerank as pg
MATCH (l)-[r:SERVES_ON]->(c:Committee)
MATCH (c)<-[:REFERRED_TO]-(:Bill)-[d:DEALS_WITH]->(s:Subject)
WHERE s.title = "Hong Kong"
MATCH (l)-[:IS_MEMBER_OF]->(p:Party)
WITH l.firstName + " " + l.lastName as name, pg, c.name AS committee,
   r.rank as rank, p.name as party
RETURN name, pg, committee, rank, party ORDER BY pg DESC LIMIT 10;
```

```md
+-------------------------------------------------------------------------------------+
| name                | pg    | committee                       | rank | party        |
+-------------------------------------------------------------------------------------+
| "Christopher Smith" | 4.79  | "House Committee on Foreign Af" | "2"  | "Republican" |
| "Edward Royce"      | 2.66  | "House Committee on Foreign Af" | "1"  | "Republican" |
| "Reid Ribble"       | 2.36  | "House Committee on Foreign Af" | "22" | "Republican" |
| "Ted Poe"           | 2.30  | "House Committee on Foreign Af" | "8"  | "Republican" |
| "Matt Salmon"       | 2.12  | "House Committee on Foreign Af" | "9"  | "Republican" |
| "David Cicilline"   | 1.92  | "House Committee on Foreign Af" | "10" | "Democrat"   |
| "Ted Yoho"          | 1.71  | "House Committee on Foreign Af" | "19" | "Republican" |
| "Joe Wilson"        | 1.60  | "House Committee on Foreign Af" | "6"  | "Republican" |
| "Steve Chabot"      | 1.55  | "House Committee on Foreign Af" | "5"  | "Republican" |
| "Tom Marino"        | 1.51  | "House Committee on Foreign Af" | "11" | "Republican" |
+-------------------------------------------------------------------------------------+
10 rows
79 ms
```

Here we can see Legislators who have influence over the subject "Hong Kong" through their membership on the House Committee on Foreign Affairs, sorted by PageRank. We should also take into account their rank on the Committee, a higher rank being another sign of a higher level of influence. We can run this query for any subject in our graph, just change the `s.title = xxx` parameter and find Congressional influencers!

#### Multiple dimensions of influences

Of course a Legislator might serve on multiple Committees that have jurisdiction over a certain subject. Here we examine Legislators who have influence over the subject "News media and reporting" and order the results by the number of Committees on which they serve that have jurisdiction over the subject.

```cypher
MATCH (l:Legislator) WHERE has(l.pagerank) WITH l, l.pagerank as pg
MATCH (l)-[r:SERVES_ON]->(c:Committee)<-[:REFERRED_TO]-(:Bill)-[d:DEALS_WITH]->(s:Subject)
WHERE s.title = "News media and reporting"
MATCH (l)-[:IS_MEMBER_OF]->(p:Party)
WITH l.firstName + " " + l.lastName as name, pg, count(d) as num, p.name as party, c, r
WITH collect({committee:c.name, rank:r.rank}) AS committees, name, pg, num, party
RETURN name, pg, committees, num, party ORDER BY size(committees) DESC LIMIT 10;
```

![](/static/images/congressional-pagerank/news_committees.png)

To take these multiple levels of influence into account we could create some sort of composite index of influence and order our results by that index, but I'll leave that as an exercise for the reader...

If you are a social scientist, or anyone interested in working with Congressional data in legis-graph please let me know your use-case and how I can help!


This post shows how we can apply graph analytics to US Congressional data to find influential legislators in Congress. Using the Mazerunner open source graph analytics project we are able to use Apache Spark GraphX alongside Neo4j to run the PageRank algorithm on a collaboration graph of US Congress.

Congressional PageRank - Analyzing US Congress With Neo4j and Apache Spark


Interactions among members of any large organization are naturally a graph, yet the tools we use to collect and analyze data about these organizations often ignore the graphiness of the data and instead map the data into structures (such as relational databases) that make taking advantage of the relationships in the data much more difficult when it comes time to analyze the data. Collaboration networks are a perfect example. So let's focus on one of the most powerful collaboration networks in the world: the US Congress.

## Govtrack.us

> Creating government transparency and civic engagement through novel uses of technology. - [govtrack.us](https://www.govtrack.us/about)

Govtrack.us is a government transparency website that helps the general public find and track bills in the US Congress and keep track of things like a congressperon's voting record and committee activity.

From Govtrack.us:

> We bring together the status of U.S. federal legislation, voting records, congressional district maps, and more ... and make it easier to understand.

Govtrack provides many web based tools for exploring and analyzing the US Congress, including a feature I find particularly interesting called Sponsorship Analysis. You can see an example of this [here](https://www.govtrack.us/congress/members/barbara_boxer/300011). Govtrack also provides an [API and makes bulk data avilable](https://www.govtrack.us/developers). Bulk data is provided in a mix of JSON, XML, and YAML.

## Introducing legis-graph: US Congress in a graph

The goal of legis-graph is to provide a way of converting the data provided by Govtrack into a rich property-graph model that can be inserted into Neo4j for analysis or integrating with other datasets.

The code and instructions are available [in this Github repo](https://github.com/legis-graph/legis-graph). The ETL workflow works like this:

1. A shell script is used to rsync data from Govtrack for a specific Congress (i.e. the 114th Congress). The Govtrack data is a mix of JSON, CSV, and YAML files. It includes information about legislators, committees, votes, bills, and much more.
1. To extract the pieces of data we are interested in for legis-graph a series of Python scripts are used to extract and transform the data from different formats into a series of flat CSV files.
1. The third component is a series of Cypher statements that make use of [`LOAD CSV`]() to efficiently import this data into Neo4j.

To get started with legis-graph in Neo4j you can follow the instructions [here](https://github.com/legis-graph/legis-graph#load-data). Alternatively, a Neo4j data store dump is available [here](https://github.com/legis-graph/legis-graph#quickstart) for use without having to execute the import scripts. We are currently working to streamline the ETL process so this may change in the future, but any updates will be appear in the [Github README](https://github.com/legis-graph/legis-graph).

This project began during preparation for a graph database focused Meetup presentation. [George Lesica](http://goto10line.net/) and I wanted to provide an interesting dataset in Neo4j for new users to explore.

## The Data Model

![](/static/images/legis-graph-congressional-data-using-neo4j/lgdatamodel.png)

As you can see from the diagram above, the datamodel is rich and captures quite a bit of information about the actions of legislators, committees, and bills in Congress. Information about what properties are currently included is available [here](https://github.com/legis-graph/legis-graph/blob/master/schema.md).

## Querying legis-graph

I recently moved to California and realized I know pretty much *nothing* about politics in California (other than the exploits of the [Governator](https://en.wikipedia.org/wiki/Arnold_Schwarzenegger), of course), so I thought a good test of legis-graph would be getting started with learning about California's Congresspeople.


**Who represents California in Congress?**

An easy start, tell me the names of California's elected representatives in Congress.

```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
RETURN l.firstName + " " + l.lastName as Legislator;
```

```md
+-------------------------+
| Legislator              |
+-------------------------+
| "Tony Cárdenas"         |
| "Raul Ruiz"             |
| "Mark Takano"           |
| "Alan Lowenthal"        |
| "Juan Vargas"           |
| "Xavier Becerra"        |
| "Karen Bass"            |
| "Barbara Boxer"         |
| "Dianne Feinstein"      |
| "Jerry McNerney"        |
| "Tom McClintock"        |
| "Kevin McCarthy"        |
| "Doris Matsui"          |
| "Zoe Lofgren"           |
| "Barbara Lee"           |
| "Loretta Sanchez"       |
| "Adam Schiff"           |
| "Lucille Roybal-Allard" |
| "Edward Royce"          |
| "Dana Rohrabacher"      |
| "Nancy Pelosi"          |
| "Grace Napolitano"      |
| "Devin Nunes"           |
| "Anna Eshoo"            |
| "Jim Costa"             |
| "Jeff Denham"           |
| "Susan Davis"           |
| "Judy Chu"              |
| "Ken Calvert"           |
| "Lois Capps"            |
| "Duncan Hunter"         |
| "Darrell Issa"          |
| "Michael Honda"         |
| "Sam Farr"              |
| "John Garamendi"        |
| "Steve Knight"          |
| "Mark DeSaulnier"       |
| "Ted Lieu"              |
| "Pete Aguilar"          |
| "Mimi Walters"          |
| "Norma Torres"          |
| "Brad Sherman"          |
| "Jackie Speier"         |
| "Linda Sánchez"         |
| "Mike Thompson"         |
| "Maxine Waters"         |
| "Janice Hahn"           |
| "David Valadao"         |
| "Julia Brownley"        |
| "Paul Cook"             |
| "Eric Swalwell"         |
| "Jared Huffman"         |
| "Ami Bera"              |
| "Doug LaMalfa"          |
| "Scott Peters"          |
+-------------------------+
55 rows
83 ms
```

OK, so California has lots of members of Congress. How many are from which parties?

**Breakdown by party**

```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[r:IS_MEMBER_OF]->(p:Party)
WITH p.name as party, count(r) as num
RETURN party, num;
```

```md
+--------------------+
| party        | num |
+--------------------+
| "Republican" | 14  |
| "Democrat"   | 41  |
+--------------------+
2 rows
56 ms
```

So California is definitely a blue state...

We can also examine this data using the Neo4j Browser.

Let's visualize California's elected representatives in both bodies of Congress:


```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator) WHERE has(l.wikipediaID)
MATCH (l)-[:ELECTED_TO]->(b:Body)
RETURN s,l,b
```

![](/static/images/legis-graph-congressional-data-using-neo4j/lg_senators_reps.png)

We can include Party in the visual as well:

```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(b:Body)
MATCH (l)-[:IS_MEMBER_OF]->(p:Party)
RETURN s,l,b,p
```

![](/static/images/legis-graph-congressional-data-using-neo4j/lg_body_party_state.png)


**Who are the Senators?**

Let's limit our analysis to only Senators representing California:

```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(b:Body {type: "Senate"})
MATCH (l)-[:IS_MEMBER_OF]->(p:Party)
RETURN s,l,b,p
```

![](/static/images/legis-graph-congressional-data-using-neo4j/lg_state_senators.png)


**What Committees do the California senators serve on?**

Each legislator serves on one or more [congressional committees](https://en.wikipedia.org/wiki/United_States_congressional_committee). In addition to other administrative powers, each committee has jurisdiction over bills of a given subject matter. By serving on certain committees legislators can gain more influence over bills that fall in their committees' jurisdictions. For example, legislators who serve on the House Committee on Agriculture have a greater influence over agricultural policies and federal funding available for agricultural programs because of the power of their committee. Examining committee membership is an initial method to examine the areas where a legislator has more influence over policy.

From our datamodel we see that we have a `SERVES_ON` relationship to determine which Committees our Senators are serving.

```cypher
MATCH (ca:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(senate:Body {type: "Senate"})
MATCH (l)-[:SERVES_ON]->(c:Committee)
RETURN l,c
```

![](/static/images/legis-graph-congressional-data-using-neo4j/ls_legislator_committees.png)


**What kinds of bills are overseen by these committees?**

First, let's examine the Bills that were referred to a single committee on which Diane Feinstein serves, the Senate Committee on Appropriations. We can do this by matching on the `thomasID` property on the Committee node.

```cypher
MATCH (ca:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(senate:Body {type: "Senate"})
MATCH (l)-[:SERVES_ON]->(c:Committee {thomasID: "SSAP"})
MATCH (c)<-[:REFERRED_TO]-(b:Bill)
RETURN ca, senate, l, c, b
```

![](/static/images/legis-graph-congressional-data-using-neo4j/lg_bills_committees.png)

So we can see 11 Bills that have been referred to the Senate Committee on Appropriations. We have information about each Bill in the properties of the node, including the title and description of the Bill, whether it was vetoed or enacated. We could also follow `VOTED_ON` relationships to see how individual Legislators voted on each Bill.

However, I'm interested in learning more about the subjects these Bills are dealing with. What I want to know is the answer to this question: "For Legislators representing California, what subjects might they have influence over based on the Committees on which they serve".

Here is just a subset of those subjects for California's Senators:

```cypher
MATCH (ca:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(senate:Body {type: "Senate"})
MATCH (l)-[:SERVES_ON]->(c:Committee)
MATCH (c)<-[:REFERRED_TO]-(b:Bill)
MATCH (b)-[:DEALS_WITH]->(s:Subject)
RETURN ca, senate, l, c, b, s
```

![](/static/images/legis-graph-congressional-data-using-neo4j/lg_committees_bills_subjects.png)

We can draw more insight from the data if we count the number of bills referred to these Committees for each subject.

```cypher
MATCH (s:State {code: "CA"})<-[:REPRESENTS]-(l:Legislator)
MATCH (l)-[:ELECTED_TO]->(b:Body) WHERE b.type="Senate"
MATCH (l)-[:SERVES_ON]-(c:Committee)<-[:REFERRED_TO]-(bill:Bill)
MATCH (bill)-[r:DEALS_WITH]->(subject:Subject)
WITH count(r) as num, subject
RETURN subject.title AS subject, num ORDER BY num DESC LIMIT 25;
```

```md
+------------------------------------------------------------+
| subject                                              | num |
+------------------------------------------------------------+
| "Crime and law enforcement"                          | 100 |
| "Congressional oversight"                            | 75  |
| "Environmental protection"                           | 66  |
| "Government information and archives"                | 54  |
| "Administrative law and regulatory procedures"       | 53  |
| "International affairs"                              | 49  |
| "Government operations and politics"                 | 40  |
| "Civil actions and liability"                        | 35  |
| "State and local government operations"              | 34  |
| "Government studies and investigations"              | 32  |
| "Law enforcement administration and funding"         | 32  |
| "Criminal procedure and sentencing"                  | 31  |
| "Criminal investigation, prosecution, interrogation" | 31  |
| "Executive agency funding and structure"             | 29  |
| "Environmental Protection Agency (EPA)"              | 29  |
| "Immigration"                                        | 29  |
| "Environmental regulatory procedures"                | 27  |
| "Water quality"                                      | 26  |
| "Criminal justice information and records"           | 25  |
| "Transportation and public works"                    | 24  |
| "Intergovernmental relations"                        | 23  |
| "Wildlife conservation and habitat protection"       | 23  |
| "Administrative remedies"                            | 22  |
| "Crime victims"                                      | 21  |
| "Licensing and registrations"                        | 21  |
+------------------------------------------------------------+
25 rows
293 ms
```

This has been a brief overview of some of the data available in [legis-graph](https://github.com/legis-graph/legis-graph). Much more complex analysis is possible and I hope those interested will find this project useful. If you find any errors or want to request changes to the data model, please submit an issue on the project's [Github page](https://github.com/legis-graph/legis-graph/issues).

Interactions among members of any large organization are naturally a graph, yet the tools we use to analyze data about these organizations often ignore the graphiness of the data and instead map the data into structures (such as relational databases) that make taking advantage of the relationships in the data much more difficult when it comes time to analyze the data. Collaboration networks are a perfect example. So we focus on one of the most powerful collaboration networks in the world, the US Congress.

Introducing legis-graph - US Congressional Data With Govtrack and Neo4j


Mining word associations from a body of text is often one of the first Natural Language Processing techniques used when mining text data. Word associations are useful for performing NLP tasks such as part of speech tagging, parsing, entity extraction, etc. We will take a brief look at one type of word association called paradigmatic association and show how we can use the Neo4j graph database to help model our text corpus as a graph and implement a simple paradigmatic relation mining algorithm.

## Mining word associations

There are two common types of word associations in natural language processing, paradigmatic and systagmatic:

* **Paradigmatic:** words A and B are paradigmatically related if they can be substituted for each other. This indicates they belong in the same class, such as "Monday" and "Thursday" or "Cat" and "Dog.
* **Syntagmatic:** words that can be combined with each other, such as "cold" and "weather".

For this post we will only concern ourselves with paradigmatic associations.

## Computing paradigmatic similarity

The basic steps to compute paradigmatic relations:

1. Represent each word by its context
1. Compute context similarity
1. Words with high context similarity likely have paradigmatic relation

### 1) Represent each word by its context

Consider a simple text document that consists of sentences like this:

```md
    My cat eats fish on Saturday.
    His dog eats turkey on Tuesday.
    ...
```

What words occur within similar context in each sentence? How could we answer this question? We need some way to represent the context for a given word. We do this by defining functions to compute pieces of the context of a given word:

```md
    Right1("cat") = {"eats", "ate", "is", "has", ...}
    Left1("cat") = {"my", "his", "big", "a", "the", ...}
```

This allows us to represent word context as a series of sets of words.

### 2) Compute context similarity

To compute a measure of paradigmatic relation we take an aggregation of the similarity of relative contexts. For example:

```md
    Sim("Cat", "Dog") =
        Sim(Left1("Cat"), Left1("Dog")) +
        Sim(Right1("Cat"), Right1("Dog"))
```

We can use the [Jaccard index](https://en.wikipedia.org/wiki/Jaccard_index) as a measure of similarity.

### 3) Words with high context similarity likely have paradigmatic relation

Once we have computed this measure of similarity we can simply look for word pairs that have a high measure of similarity.

> **NOTE:** this is a relatively simplistic approach. This method could be imporoved by expanding the size of the context window (for example, including `Left2`, `Window2` word context sets), handling [stop words](https://en.wikipedia.org/wiki/Stop_words), and term frequency based transformations to adjust similarity scores.

## Modeling text data as an adjacency graph

We can model text data as an adjacency graph where each word is a node and an edge between two nodes indicates that the words appear next to each other in the text corpus. The data model we will use looks like this:

![](/static/images/nlp-with-neo4j/nlp_model.png)

As an example, consider the following sentence:

    My cat eats fish on Saturday.

Using our data model this would appear as follows in our adjacency graph:

![](/static/images/nlp-with-neo4j/first_sentence.png)

If we add another sentence:

    His dog eats turkey on Tuesday.

Our graph is updated like this:

![](/static/images/nlp-with-neo4j/second_sentence.png)

We will use the Neo4j graph database to implement this data model. The text corpus used here is the [CEEAUS](http://earthlab.uoi.gr/theste/index.php/theste/article/viewFile/55/37) [corpus](http://language.sakura.ne.jp/s/eng.html) which is distributed with the [MeTA NLP library](https://meta-toolkit.org/).

We will first see how we can load this text corpus into Neo4j using a Python script. Then we will explore how we can query the data to mine paradigmatic relations using Cypher, the Neo4j query language.

## Loading the data

The data is stored in a single file, with each sentence on a single line. We define a Python function to load this file, normalize the text a bit and execute a Cypher query for each line of text to insert into the graph as per the data model we have defined above.  Here's the Python code for data insertion:

```python
from py2neo import Graph
import re, string

# default uri for local Neo4j instance
graphdb = Graph('http://neo4j:neo4j@localhost:7474/db/data')

# parameterized Cypher query for data insertion
# t is a query parameter. a list with two elements: [word1, word2]
INSERT_QUERY = '''
    FOREACH (t IN {wordPairs} |
        MERGE (w0:Word {word: t[0]})
        MERGE (w1:Word {word: t[1]})
        CREATE (w0)-[:NEXT_WORD]->(w1)
        )
'''

# convert a sentence string into a list of lists of adjacent word pairs
# also convert to lowercase and remove punctuation using a regular expression
# arrifySentence("Hi there, Bob!) = [["hi", "there"], ["there", "bob"]]
def arrifySentence(sentence):
    sentence = sentence.lower()
    sentence = sentence.strip()
    exclude = set(string.punctuation)
    regex = re.compile('[%s]' % re.escape(string.punctuation))
    sentence = regex.sub('', sentence)
    wordArray = sentence.split()
    tupleList = []
    for i, word in enumerate(wordArray):
        if i+1 == len(wordArray):
            break
        tupleList.append([word, wordArray[i+1]])
    return tupleList

# load our text corpus into Neo4j
def loadFile():
    tx = graphdb.cypher.begin()
    with open('data/ceeaus.dat', encoding='ISO-8859-1') as f:
        count = 0
        for l in f:
            params = {'wordPairs': arrifySentence(l)}
            tx.append(INSERT_QUERY, params)
            tx.process()
            count += 1
            # process in batches of 100 insertion queries
            if count > 100:
                tx.commit()
                tx = graphdb.cypher.begin()
                count = 0
    f.close()
    tx.commit()
```

## Mining word relations

Now that we've inserted our data we need some way to query it to discover word relations.

### A Cypher / Python Approach

We can write a simple Cypher query to find the `Right1` and `Left1` sets quite easily. We can also define some Python methods to handle these Cypher queries and perform some simple set operations to compute Jaccard similarity for a word pair:

```python
# get the set of words that appear to the left of a specified word in the text corpus
LEFT1_QUERY = '''
    MATCH (s:Word {word: {word}})
    MATCH (w:Word)-[:NEXT_WORD]->(s)
    RETURN w.word as word
'''

# get the set of words that appear to the right of a specified word in the text corpus
RIGHT1_QUERY = '''
    MATCH (s:Word {word: {word}})
    MATCH (w:Word)<-[:NEXT_WORD]-(s)
    RETURN w.word as word
'''

# return a set of all words that appear to the left of `word`
def left1(word):
    params = {
        'word': word.lower()
    }
    tx = graphdb.cypher.begin()
    tx.append(LEFT1_QUERY, params)
    results = tx.commit()
    words = []
    for result in results:
        for line in result:
            words.append(line.word)
    return set(words)

# return a set of all words that appear to the right of `word`
def right1(word):
    params = {
        'word': word.lower()
    }
    tx = graphdb.cypher.begin()
    tx.append(RIGHT1_QUERY, params)
    results = tx.commit()
    words = []
    for result in results:
        for line in result:
            words.append(line.word)
    return set(words)

# compute Jaccard coefficient
def jaccard(a,b):
    intSize = len(a.intersection(b))
    unionSize = len(a.union(b))
    return intSize / unionSize

# we define paradigmatic similarity as the average of the Jaccard coefficents of the `left1` and `right1` sets
def paradigSimilarity(w1, w2):
    return (jaccard(left1(w1), left1(w2)) + jaccard(right1(w1), right1(w2))) / 2.0
```

This allows us to compute a measure of Paradigmatic similarity for arbitrary word pairs:

    In [174]: paradigSimilarity("school", "university")
    Out[174]: 0.2153846153846154

### A Pure Cypher Approach

The Cypher / Python approach above doesn't really allow us to identify the most paradigmatically similar words. For example, we really only can ask "what is the paradigmatic similarity between words x,y". What we really want to ask is "what are the word pairs that are most paradigmatically similar?" To do this we will use a pure Cypher approach to compare word pairs and update the graph with a `:RELATED_TO` relationship. This relationship will store the similarity score. Once this query runs we can query for word relations using this new `:RELATED_TO` edge.

Here's the query:

```cypher
MATCH (s:Word)
// Get right1, left1
MATCH (w:Word)-[:NEXT_WORD]->(s)
WITH collect(DISTINCT w.word) as left1, s
MATCH (w:Word)<-[:NEXT_WORD]-(s)
WITH left1, s, collect(DISTINCT w.word) as right1
// Match every other word
MATCH (o:Word) WHERE NOT s = o
WITH left1, right1, s, o
// Get other right, other left1
MATCH (w:Word)-[:NEXT_WORD]->(o)
WITH collect(DISTINCT w.word) as left1_o, s, o, right1, left1
MATCH (w:Word)<-[:NEXT_WORD]-(o)
WITH left1_o, s, o, right1, left1, collect(DISTINCT w.word) as right1_o
// compute right1 union, intersect
WITH FILTER(x IN right1 WHERE x IN right1_o) as r1_intersect,
  (right1 + right1_o) AS r1_union, s, o, right1, left1, right1_o, left1_o
// compute left1 union, intersect
WITH FILTER(x IN left1 WHERE x IN left1_o) as l1_intersect,
  (left1 + left1_o) AS l1_union, r1_intersect, r1_union, s, o
WITH DISTINCT r1_union as r1_union, l1_union as l1_union, r1_intersect, l1_intersect, s, o
WITH 1.0*length(r1_intersect) / length(r1_union) as r1_jaccard,
  1.0*length(l1_intersect) / length(l1_union) as l1_jaccard,
  s, o
WITH s, o, r1_jaccard, l1_jaccard, r1_jaccard + l1_jaccard as sim
CREATE UNIQUE (s)-[r:RELATED_TO]->(o) SET r.paradig = sim;
```

We iterate over word pairs, fetch the `right1` and `left1` word sets, then perform the equivilent of union and intersection operations that allow us to compute the Jaccard coefficient for each word pair. Finally, we update the graph with this coefficent in a new `:RELATED_TO` relationship.

## Querying for word relations

Now that we've updated the graph with these word association scores we can write simple Cypher queries to discover word associations.

### Paradigmatic relations for "school"

What are the words most strongly paradigmatically related to the word "school"?

```cypher
MATCH (s:Word {word: 'school'} )-[r:RELATED_TO]->(o) 
RETURN s.word,o.word,r.paradig as sim ORDER BY sim DESC LIMIT 25;
```

```md
+-----------------------------------------------+
| s.word   | o.word       | sim                 |
+-----------------------------------------------+
| "school" | "university" | 0.35416666666666663 |
| "school" | "studies"    | 0.35279106858054227 |
| "school" | "working"    | 0.3497129735935706  |
| "school" | "parents"    | 0.33613445378151263 |
| "school" | "college"    | 0.33519813519813524 |
| "school" | "society"    | 0.3310029130253849  |
| "school" | "parttime"   | 0.3220510229856024  |
| "school" | "study"      | 0.3186426358772177  |
| "school" | "money"      | 0.31618037135278515 |
| "school" | "eating"     | 0.31335453100158983 |
| "school" | "place"      | 0.3115313895455739  |
| "school" | "experience" | 0.30545975141537063 |
| "school" | "work"       | 0.3031372549019608  |
| "school" | "life"       | 0.30304354157565166 |
| "school" | "smoker"     | 0.29819694868238555 |
| "school" | "jobs"       | 0.2965686274509804  |
| "school" | "living"     | 0.29284676833696444 |
| "school" | "future"     | 0.2874499332443258  |
| "school" | "hard"       | 0.28233938346297893 |
| "school" | "children"   | 0.2812025508654722  |
| "school" | "studying"   | 0.27817343946376205 |
| "school" | "food"       | 0.27809596538118875 |
| "school" | "restaurant" | 0.2743995912110373  |
| "school" | "meal"       | 0.2705982542048116  |
| "school" | "friends"    | 0.2692307692307693  |
+-----------------------------------------------+
25 rows
45 ms
```

### Strong paradigmatic relation

What are the strongest paradigmatically related words in the text corpus?

```cypher
MATCH (s)-[r:RELATED_TO]->(o) 
RETURN s.word,o.word,r.paradig AS sim ORDER BY sim DESC LIMIT 100;
```

```md
+---------------------------------------------------+
| s.word      | o.word        | sim                 |
+---------------------------------------------------+
| "opinion"   | "idea"        | 0.39954526663910706 |
| "a"         | "the"         | 0.39918436823602943 |
| "students"  | "college"     | 0.393037294302489   |
| "students"  | "people"      | 0.3798992202990593  |
| "students"  | "smokers"     | 0.37755673858223593 |
| "in"        | "for"         | 0.37120642515379354 |
| "for"       | "in"          | 0.37120642515379354 |
| "important" | "hard"        | 0.37101588316810574 |
| "opinion"   | "area"        | 0.36708144796380093 |
| "opinion"   | "situation"   | 0.3667582417582418  |
| "important" | "necessary"   | 0.3656898656898657  |
| "important" | "bad"         | 0.35802139037433156 |
| "school"    | "university"  | 0.35416666666666663 |
| "students"  | "you"         | 0.35373408158860353 |
| "important" | "harmful"     | 0.3533992213237496  |
| "school"    | "studies"     | 0.35279106858054227 |
| "for"       | "of"          | 0.35274323661808615 |
| "to"        | "and"         | 0.35257738943248285 |
| "school"    | "working"     | 0.3497129735935706  |
| "students"  | "restaurants" | 0.3489169719469919  |
| "important" | "difficult"   | 0.3485156912637829  |
| "students"  | "student"     | 0.34606662818504164 |
| "for"       | "that"        | 0.34368031605589777 |
| "students"  | "parents"     | 0.33853521126760566 |
| "opinion"   | "place"       | 0.3381635025184037  |
| "for"       | "to"          | 0.33750185739002114 |
| "to"        | "for"         | 0.33750185739002114 |
| "students"  | "nonsmokers"  | 0.33671364061040177 |
| "school"    | "parents"     | 0.33613445378151263 |
| "school"    | "college"     | 0.33519813519813524 |
| "opinion"   | "fact"        | 0.3350840336134454  |
| "opinion"   | "reasons"     | 0.3310381355932204  |
| "school"    | "society"     | 0.3310029130253849  |
| "students"  | "money"       | 0.3286558659692988  |
| "have"      | "do"          | 0.3284722920647692  |
| "for"       | "but"         | 0.32701345186452413 |
| "important" | "reason"      | 0.3252759802055577  |
| "for"       | "by"          | 0.3252038356724162  |
| "my"        | "our"         | 0.3236480567977502  |
| "important" | "good"        | 0.3223516563523844  |
| "school"    | "parttime"    | 0.3220510229856024  |
| "for"       | "if"          | 0.32164442810636096 |
| "my"        | "their"       | 0.32032052127726696 |
| "opinion"   | "statement"   | 0.3194444444444444  |
| "opinion"   | "problem"     | 0.31904269972451793 |
| "opinion"   | "point"       | 0.31894736842105265 |
| "school"    | "study"       | 0.3186426358772177  |
| "students"  | "restaurant"  | 0.31855802425422675 |
| "students"  | "work"        | 0.3179591836734694  |
| "school"    | "money"       | 0.31618037135278515 |
| "opinion"   | "lives"       | 0.3159090909090909  |
| "opinion"   | "day"         | 0.3156288156288156  |
| "in"        | "and"         | 0.3154907057525759  |
| "in"        | "of"          | 0.3153160870310091  |
| "in"        | "at"          | 0.31521761006289306 |
| "for"       | "and"         | 0.3140333542277307  |
| "opinion"   | "case"        | 0.3134920634920635  |
| "school"    | "eating"      | 0.31335453100158983 |
| "have"      | "get"         | 0.312328440147989   |
| "school"    | "place"       | 0.3115313895455739  |
| "opinion"   | "country"     | 0.3101851851851852  |
| "to"        | "that"        | 0.30992732204777806 |
| "important" | "studying"    | 0.30947775628626695 |
| "students"  | "them"        | 0.3093731519810763  |
| "for"       | "so"          | 0.3077891508736062  |
| "students"  | "smoker"      | 0.30633078730904817 |
| "students"  | "us"          | 0.30616821341202966 |
| "important" | "well"        | 0.3059533175976921  |
| "school"    | "experience"  | 0.30545975141537063 |
| "opinion"   | "workplace"   | 0.3048855905998763  |
| "school"    | "work"        | 0.3031372549019608  |
| "school"    | "life"        | 0.30304354157565166 |
| "opinion"   | "reason"      | 0.30277349768875195 |
| "for"       | "from"        | 0.3025785639266867  |
| "have"      | "make"        | 0.3020345644760418  |
| "students"  | "working"     | 0.3019127064374789  |
| "students"  | "smoking"     | 0.3011066418324483  |
| "for"       | "on"          | 0.3011051106321491  |
| "for"       | "as"          | 0.30039366281771396 |
| "in"        | "to"          | 0.300318818247031   |
| "to"        | "in"          | 0.300318818247031   |
| "opinion"   | "space"       | 0.3003003003003003  |
| "students"  | "society"     | 0.300220818254309   |
| "opinion"   | "condition"   | 0.3001605136436597  |
| "for"       | "with"        | 0.29984269706562056 |
| "in"        | "but"         | 0.2997641267562301  |
| "school"    | "smoker"      | 0.29819694868238555 |
| "students"  | "time"        | 0.29796635538346405 |
| "opinion"   | "rule"        | 0.2975825672454886  |
| "opinion"   | "matter"      | 0.2975543478260869  |
| "important" | "precious"    | 0.2971576227390181  |
| "in"        | "that"        | 0.2970460614152203  |
| "important" | "place"       | 0.2969177779740351  |
| "important" | "dangerous"   | 0.29680644832159986 |
| "important" | "much"        | 0.2967315570706618  |
| "school"    | "jobs"        | 0.2965686274509804  |
| "have"      | "be"          | 0.296095994591854   |
| "opinion"   | "meal"        | 0.29527272727272724 |
| "opinion"   | "way"         | 0.2952157388777107  |
| "to"        | "of"          | 0.29506851814362445 |
+---------------------------------------------------+
100 rows
425 ms
```

All Python code is available [here](https://github.com/johnymontana/nlp-graph-notebooks/blob/master/paradigmatic-relations.ipynb).

**Final note:** Many of the NLP ideas and examples here were borrowed from the Coursera course [Text Mining and Analytics](https://class.coursera.org/textanalytics-001) by [Dr. ChengXiang Zhai](http://czhai.cs.illinois.edu/).

Discovering paradigmatic word associations is one of the core tasks of Natural Language Processing. In this post we explore a graph based method for mining paradigmatic word associations from a text corpus.

Natural Language Processing With Neo4j - Mining Paradigmatic Word Associations

Using article links shared on Twitter to determine user interests, we show how to build a personalized content recommender system to suggest other articles the user might be interested in.

Content Recommendation From Links Shared on Twitter Using Neo4j and Python


![](/static/images/using-neo4j-spatial-and-leaflet-js-with-mapbox/screenshot.png)

A common use-case for database queries is to search for things that are close to other things or within some specified geospatial boundary. Geospatial indexes and queries are offered by NoSQL databases, such as [MongoDB](http://docs.mongodb.org/manual/core/geospatial-indexes/) and relational databases such as [PostgreSQL](http://postgis.net/). But what about graph databases? In this article, I show how to create a web application to search within a user-defined boundary powered by the Neo4j graph database.

## Searching for businesses within a user defined boundary

Our requirements for this web app are:

1. allow the user to draw a polygon on a map
1. allow the user to select the category of business they would like to search for
1. show the user businesses that match their selected category and within the bounds they have defined on the map

You can check it out [here](http://spatialcypherdemo.herokuapp.com/), but this is what it looks like:

![spatial query gif](/static/images/using-neo4j-spatial-and-leaflet-js-with-mapbox/out.gif)

In the rest of this post we'll look at how to build this project.

## The stack

Let's take a look at the components of this project.

#### Data

The data for this project comes from the [Yelp Academic Dataset](https://www.yelp.com/academic_dataset). At the time that I built this project I was a graduate student at the University of Montana so I'm pretty sure I was operating within the confines of Yelp's license for the use of this data. Needless to say, I can't redistribute the data so if you'd like to follow along you'll need to download it directly from Yelp.

The dataset contains information about over 100,000 businesses in the Phoenix, AZ area in the U.S. This includes Yelp user reviews and some anonymized information about the users. We will use the latitude / longitude data to map businesses, as well as the business category for filtering.

#### Neo4j Spatial

[Neo4j](http://neo4j.com) is a graph database that allows for modeling, storing and querying data as a graph. If you haven't been exposed to graph databases yet it's worth checking out as many use cases are naturally modeled as a graph. [Neo4j Spatial](https://github.com/neo4j-contrib/spatial) is a plugin for Neo4j that enables spatial operations: spatial indexing and spatial querying (such as find things within x distance of y or find things within a certain geometry.

Here we will use Neo4j Spatial's `SearchIntersects` query to perform an efficent spatial query to find businesses within a user defined polygon.

#### MapBox / Leaflet.js
[Leaflet.js](http://leafletjs.com/) is an open-source JavaScript maps library that makes it really easy to create maps in the browser. MapBox provides an extension to Leaflet, Mapbox.js, that adds some features and makes it easier to work with MapBox's tile servers.

Now that we've seen what tools we'll be working with, the first step is to load our data into Neo4j.

## Loading the data

The data consists of an array of JSON objects describing each business. Something like this:

```js
// from https://www.yelp.com/academic_dataset

{
  'type': 'business',
  'business_id': (a unique identifier for this business),
  'name': (the full business name),
  'neighborhoods': (a list of neighborhood names, might be empty),
  'full_address': (localized address),
  'city': (city),
  'state': (state),
  'latitude': (latitude),
  'longitude': (longitude),
  'stars': (star rating, rounded to half-stars),
  'review_count': (review count),
  'photo_url': (photo url),
  'categories': [(localized category names)]
  'open': (is the business still open for business?),
  'schools': (nearby universities),
  'url': (yelp url)
}
```

Lots of interesting data available here but we are only interested in `business_id`, `name`, `latitude`, `longitude`, and `categories`.

There are [several](http://neo4j.com/docs/stable/query-load-csv.html) [ways](http://neo4j.com/docs/stable/import-tool.html) to import data into Neo4j. The Neo4j Spatial plugin even provides functionality for importing from [shapefiles and from OSM data](https://github.com/neo4j-contrib/spatial#examples). I hadn't originally considered making use of the business category information so I ended up with a messy two step data import process:

1. Create a Business node for each business in the dataset and add it to the spatial index
1. Create relationships between this Business node and the appropriate Category nodes

Including the in-graph spatial index, the data model looks like this:

![](/static/images/using-neo4j-spatial-and-leaflet-js-with-mapbox/scdemo_datamodel.png)

##### Create Business nodes and add to spatial index

I chose to create an [unmanaged server extension](http://neo4j.com/docs/stable/server-unmanaged-extensions.html) for interacting with Spatial using the Java API (more on this in the next section), adding an endpoint for Business node creation. This endpoint takes a JSON object that represents the business to be created and added to the spatial index:

```json
{
    "business_id": "lajsldkjfaslj48o",
    "name": "Bob's Burgers",
    "address": "123 Fake St. Notacity, MT 59601",
    "lat": 22.21323,
    "lon": 14.43211
}
```

I wrote a simple Python script to iterate through all business objects in the dataset and make a POST request to our new endpoint. The script is available [here](https://github.com/johnymontana/scdemo/blob/master/yelp_post.py). We should really be posting an array of business objects here, but we'll save that for v2 ;-) More in the next section about how this POST request is actually handled.

##### Add category relationships to Business nodes

I quickly realized that having business category information would make this demo a bit more interesting. Each business has an array of categories to which it belongs. So we now iterate through the dataset again, but this time execute a cypher query that will add a relationship from the Business node to the appropriate Category nodes for each business, creating our Category nodes along the way using a Cypher `MERGE` statement:

```python
import json
from py2neo import neo4j

db = neo4j.GraphDatabaseService("http://server_addr_here:7474/db/data/")

merge_category_query = '''
  MATCH (b:Business {business_id: {business_id}})
  MERGE (c:Category {name: {category}})
  CREATE UNIQUE (c)<-[:IS_IN]-(b)
'''

print "Beginning category batch"
with open('data/yelp_academic_dataset_business.json', 'r') as f:
  category_batch = neo4j.WriteBatch(db)
  count = 0
  for b in (json.loads(l) for l in f):
    for c in b['categories']:
      category_batch.append_cypher(merge_category_query, {'business_id': b['business_id'], 'category': c})
      count += 1
      if count >= 10000:
        category_batch.run()
        category_batch.clear()
        print "Running batch"
        count = 0
  if count > 0:
    category_batch.run()
```

This script uses the [py2neo](https://github.com/nigelsmall/py2neo) Python library written by [Nigel Small](https://twitter.com/neonige) that allows for easy Neo4j interaction from Python.



## Neo4j Spatial queries / server extension

As I mentioned, we use a Neo4j server extension to add two endpoints to the Neo4j REST API. The first endpoint handles a POST request with the data for a business and then creates a corresponding Business node in the database and adds this node to our spatial index, allowing for later spatial query operations. The second endpoint will handle a GET request, with a [polygon WKT](http://en.wikipedia.org/wiki/Well-known_text) string and business category as parameters and return an array of all businesses found within the polygon that match the given category.





#### `POST .../scdemo/node`
The purpose of this endpoint is to add a given Business into the Spatial layer. Unmanaged extensions allow us to define arbitrary [JAX-RS](http://en.wikipedia.org/wiki/Java_API_for_RESTful_Web_Services) handlers to be executed when a request is sent to our new endpoints. We saw above that we made a series of POST requests, sending a JSON body with details for each business. Here we will look at the code for the handler for that endpoint.

Handler to create Business node and add to the Spatial index:

```java
@POST
@Path("/node")
public Response addNode(String nodeParamsJson, @Context GraphDatabaseService db) {
  Node businessNode;

  SpatialDatabaseService spatialDB = new SpatialDatabaseService(db);

  Gson gson = new Gson();
  BusinessNode business = gson.fromJson(nodeParamsJson, BusinessNode.class);

  try ( Transaction tx = db.beginTx()) {
    businessNode = db.createNode();
    businessNode.addLabel(Labels.Business);
    businessNode.setProperty("business_id", business.getBusiness_id());
    businessNode.setProperty("name", business.getName());
    businessNode.setProperty("address", business.getAddresss());
    businessNode.setProperty("lat", business.getLat());
    businessNode.setProperty("lon", business.getLon());
    tx.success();
  }

  try (Transaction tx = db.beginTx()) {
    Layer businessLayer = spatialDB.getOrCreatePointLayer("business", "lat", "lon");
    businessLayer.add(businessNode);
    tx.success();
  }

  return Response.ok().build();

}
```

We first declare that this method will handle POST requests to `.../node` and that we expect a JSON body string. Next, we get a handle on the `GraphDatabaseServive` and instantiate a new `SpatialDatabaseService`. Really this should only be done once, and then cached for later calls, but I'm just trying to show each handler as a self-enclosed method. Next, we serialize the JSON body to an instance of `BusinessNode` using the [google-gson](https://github.com/google/gson) Java library. `BusinessNode` is a [POJO](http://en.wikipedia.org/wiki/Plain_Old_Java_Object) with instance vars for the properties of our business and the appropriate getters / setters to facilitate serialization with GSON. We then create a new Neo4j node called `businessNode` and set the appropriate properties on this node before committing the transaction. Finally, we get a handle of the Neo4j Spatial point layer and add this newly created `businessNode` to the Spatial layer. Spatial will take care of the appropriate in-graph R-Tree indexing and we can now make spatial query operations on the `businessNode`.

#### `GET .../scdemo/intersects`

```md
GET .../scdemo/intersects?polygon=WKTPOLYGON&category=xxx
```

The other endpoint we add in our server extension executes the spatial query. We send it a business category and a polygon and in return we get an array of all businesses within that polygon that match the business category.

Handler to query within polygon:

```java
@GET
@Path("/intersects/")
public Response getBusinessesInPolygon(@QueryParam("polygon") String polygon, @QueryParam("category") String category, @Context GraphDatabaseService db) throws IOException, ParseException{
    WKTReader wktreader = new WKTReader();

    ArrayList<Object> resultsArray = new ArrayList();

    SpatialDatabaseService spatialDB = new SpatialDatabaseService(db);
    Layer businessLayer = spatialDB.getOrCreatePointLayer("business", "lat", "lon");
    SpatialIndexReader spatialIndex = businessLayer.getIndex();

    SearchIntersect searchQuery = new SearchIntersect(businessLayer, wktreader.read(polygon));


    try (Transaction tx = db.beginTx()) {
        SearchResults results = spatialIndex.searchIndex(searchQuery);


        for (Node business : results) {
            for (Relationship catRel : business.getRelationships(RelTypes.IS_IN, Direction.BOTH)) {
                Node categoryNode = catRel.getOtherNode(business);
                if (categoryNode.getProperty("name").equals(category)) {
                    HashMap<String, Object> geojson = new HashMap<>();
                    geojson.put("lat", business.getProperty("lat"));
                    geojson.put("lon", business.getProperty("lon"));
                    geojson.put("name", business.getProperty("name"));
                    geojson.put("address", business.getProperty("address"));
                    resultsArray.add(geojson);
                }
            }

        }

        tx.success();
    }

    ObjectMapper objectMapper = new ObjectMapper();
    return Response.ok().entity(objectMapper.writeValueAsString(resultsArray)).build();
}
```

Here we define the handler of a GET request and specify two string query parameters: the polygon, as a [WKT string](http://en.wikipedia.org/wiki/Well-known_text), and the business category. We instantiate a new `SpatialDatabaseService` and get a handle on our Spatial Layer `businessLayer`. We then use Spatial's `SearchIntersect` query to perform a spatial query within our `businessLayer` to find any Nodes intersecting the polygon. We then iterate through the results and build up an array of `HashMap` objects with the business name, latitude and longitude to return as JSON.

## Mapbox / Leaflet frontend

Now that we've imported the data into Neo4j, set up our spatial index, and have an endpoint that will allow us to perform the spatial query that we need it's time to build a frontend for this application. We wil create a web app that renders a map, allows the user to draw an arbitrary polygon on the map and select a business category. Our web app will then need to send a request to our Neo4j server extension endpoint to perform the spatial query and render the results as pin annotations on the map.

##### Load the map

We will use the excellent [Mapbox.js](https://www.mapbox.com/mapbox.js/) JavaScript library to facilitate this. Mapbox.js is an extension to the [Lealet.js](http://leafletjs.com/) library that adds some functionality and makes working with Mapbox tile servers a breeze.

First we'll need to display the map, enable polygon drawing and define which functions should be called once the polygon is drawn or an existing polygon edited:

```js
L.mapbox.accessToken = 'MAPBOX_TOKEN_HERE';
var map = L.mapbox.map('map', 'MAP_ID_HERE')
    .setView([33.47870401153533, -112.0305061340332], 14);
var featureGroup = L.featureGroup().addTo(map);
var dataLayer = L.mapbox.featureLayer().addTo(map);
var current_poly;
var url = "http://NEO4j_SERVER_URL_HERE:7474/scdemo/scdemo/intersects";

// enable polygon drawing
var drawControl = new L.Control.Draw({
      edit: {
        featureGroup: featureGroup
      },
      draw: {
        polygon: true,
        polyline: false,
        rectangle: false,
        circle: false,
        marker: false
      }
    }).addTo(map);

// call showPolygonMarkers function when a polygon is created
map.on('draw:created', showPolygonMarkers);

// call showPolygonMarkersEdited when an existing polygon is edited
map.on('draw:edited', showPolygonMarkersEdited);

// if the category is changed and we have an existing polygon in the map, perform a new query to find businesses in the polygon matching the new category
$("#category").change(function() {
      if (current_poly){
        data = {polygon: current_poly, category: $("#category").val()}
        $.ajax({
          url: url,
          data: data,
          success: dataToMarkers
        });
      }
});
```

##### Handlers for polygon drawing / editing

Here we build a WKT Polygon string from the user defined polygon and make an AJAX request to our Neo4j server extension to perform the spatial query to find all businesses within our polygon / category:

```js
function showPolygonMarkersEdited(e) {
      e.layers.eachLayer(function(layer) {
        showPolygonMarkers({ layer: layer });
      });
    }
    function showPolygonMarkers(e) {
      console.log(e);
      console.log($("#category").val());
      featureGroup.clearLayers();
      featureGroup.addLayer(e.layer);

      coords = e.layer['_latlngs']
      first = coords[0]['lat'] + " " + coords[0]['lng']+", "
      wkt = "POLYGON (("
      coords.forEach(getCoords)
      wkt = wkt + first.substring(0, first.length-2) + "))"
      console.log(wkt)

      current_poly = wkt
      data = {polygon: wkt, category: $("#category").val()}
      success = function(data){
        console.log(data)
      }

      $.ajax({
        url: url,
        data: data,
        success: dataToMarkers
        //dataType: dataType
      });
    }
    function getCoords(e) {
      group = e['lat'] + " " + e['lng']+", "
      console.log(group)
      wkt = wkt + group
    }
```

##### Render the markers on Ajax success

Now we just need to create markers for each business in the JSON array that comes back from our server extension and add that layer to the map:

```js
function dataToMarkers(data) {
        data = JSON.parse(data)
        var geojson = { type: 'FeatureCollection', features: [] };

        for (var i = 0; i < data.length; i++) {

        if (data[i].lon === null || data[i].lat === null) continue;
        geojson.features.push({
          type: 'Feature',
            geometry: {
              type: 'Point',
              coordinates: [ data[i].lon, data[i].lat]
            },
          properties: {
        'marker-size': "large",
        'marker-color': "#c091e6",
        'title': data[i].name
        }
        });
      }
      dataLayer.setGeoJSON([]);
      dataLayer.setGeoJSON(geojson);
      }
```

Well that's about it. You can see the demo live [here](http://spatialcypherdemo.herokuapp.com). The full code is available on my [GitHub](http://github.com/johnymontana) page. Code for the Neo4j server extension is available [here](https://github.com/johnymontana/scdemo-extension) and code for data import and the web app is available in [this repository](https://github.com/johnymontana/scdemo).

If you'd like to be notified when I publish more posts like this, you can [follow me on Twitter](http://twitter.com/lyonwj).

As an aside, I was fortunate to receive a Google Summer of Code grant last summer to work with the folks at OSGeo and Neo Technology to build a prototype allowing for interacting with Neo4j Spatial directly from the Cypher query language. You can read more about that [here](https://github.com/johnymontana/neo4j/wiki/tutorial).


We show how to load data from the Yelp Academic Dataset into Neo4j, then build a webapp to perform geospatial queries on the data and visualize the results on a map.

Using Neo4j Spatial and Mapbox to search for businesses by location


import YouTube from '../../components/YouTube';

![Santiago, Chile](/static/images/startup-chile-week-1/IMG_1538.png)

It's been one week since I arrived in Santiago, Chile to participate in [Startup Chile](http://www.startupchile.org/) as cofounder of [Datamaglia](http://www.datamaglia.com/). Startup Chile was created four years ago as an economic stimulus experiment to boost entrepreneurism in Chile. The program provides a ~$33,000 grant and a one year temporary resident visa to startups to come work in Chile for six months. In exchange, we promise to participate in the community and help grow Chile's startup community. The program is just beginning for me and I'll be sharing some of my experiences here.

## Intro Days

![intro days](/static/images/startup-chile-week-1/IMG_1557.png)

The first week consisted of getting to know everyone in my cohort (85 startups), and learning more about the program. Startup Chile has no doubt had an impact on the startup community in Chile. Since funding its first batch four years ago more than 1000 startups have gone through the program (with participants from 75 countries), and more than 10 Chilean investment funds have been created. Examples of successful startups from previous Startup Chile rounds include [WeHostel](http://www.crunchbase.com/organization/wehostels), [BabyTuto](http://www.babytuto.com/), and [URX](http://urx.com/) (now in YCombinator).

For those interested in keeping up with news from Startup Chile check out the [SUP News YouTube channel](https://www.youtube.com/user/startupchile/videos) (with English subtitles).

![Finance Minister](/static/images/startup-chile-week-1/IMG_1563.png)

*The Chilean Finance Minister, Luis Felipe Céspedes, visits Startup Chile Intro Days to encourage us to help spread entrepreneurship to Chile's regions.*

**Pitches**

![pitch](/static/images/startup-chile-week-1/2014-11-18-10-57-03-1.png)

Each team in the cohort gave a 3 minute pitch to introduce their business and what they are working on. A full list of the startups in my cohort is available [here](http://www.startupchile.org/here-are-the-100-startups-chosen-to-be-part-of-start-up-chile-gen-11/). Most of the startups are focused on software (web and mobile apps), but also several in the biotech space and a few hardware startups.

**Coworking**

The main Startup Chile coworking space has room for 100 teams, fast internet and a beautiful glass ceiling / LCD screen covering the main room. It is located in Providencia, not far from Plaza Italia.

![](/static/images/startup-chile-week-1/IMG_1552.png)

We also have access to the administrative offices in Centro Santiago.

![](/static/images/startup-chile-week-1/2014-11-10-08-48-42.png)

*The administrative office for Startup Chile.*

### TODOs

Goals for next week and moving forward.

* **Find an apartment.** We have temporary accommodations for the next few weeks, but need to find a larger space once my cofounder (George) and his girlfriend arrive.
* **Get to work!** I'm glad to have most of the initial intro days / paperwork completed and now have time to focus on what I came to Santiago to do: grow [Datamaglia](http://datamaglia.com)!

For anyone interested in following my Santiago adventures more closely I'll be posting pictures to my new [Instagram account](http://instagram.com/johnymontana) and on [Twitter @lyonwj](https://twitter.com/lyonwj).


What have I been doing in Santiago Chile as part of Startup Chile? I share some of my experiences from Startup Chile Generation 11.

Startup Chile Week 1


![Graph connect stage photo](/static/images/notes-from-graphconnect-2014/IMG_1437.png)


# GraphConnect 2014 - San Francisco

GraphConnect is Neo Technology's annual Graph Database conference. Many of the talks highlight Neo4j use-cases, but many of the concepts are applicable to graph data modeling and graph databases in general.

## Advanced Cypher training class

The day before the conference I was able to take the Advanced Cypher training course. This course was a great in-depth dive into Cypher and I was happy that the course lived up to its name, "Advanced" Cypher. [Wes Freeman](https://twitter.com/wefreema) did a great job leading the course. Thanks, Wes!


## Graph Hack

The evening before GraphConnect we participated in Graph Hack, a hackathon focused on building Neo4j apps using a handful of open source datasets put together by [Nicole White](https://twitter.com/_nicolemargaret). The datasets consisted of flight and plane data, music playlists, and public transportation data. You can find them available [here](https://gist.github.com/nicolewhite/cc178bf2a761d7ac3a20) if you'd like to play around with the data. My team built a simple command line app that computed the probability of your flight being delayed given the origin, destination, airline, day of flying and type of aircraft.

**The following are my notes from the conference:**

## Keynote: Graphs Are Eating The World

### Emil Eifrem (CEO, Neo Technology)

![](/static/images/notes-from-graphconnect-2014/IMG_1439.png)


* The internet of **connected** things
* Graphs are eating the world
* Graph DBs are fastest growing NoSQL segment
* Example social query
	* average 50 friends, see if two random users are connected
	* 2ms Neo4j (1000 or 1000000) users
	* 2s with MySQL (just 1000 friends)
* Neo4j 2.2 - performance and scale
	* initial import performance - millions of inserts per second
	* cypher performance improvements
	* concurrent writes - ACID write to disk (currently sequential)
		* batch / concurrent writes is coming in 2.2
		* 100x faster for large writes

## The Business Graph

### Kurt Freytag (Head of Product, CrunchBase )

* CrunchBase started in 2007 with 2 crappy AWS servers (1 MySQL, 1 Rails)
* Power of crowd sourced data
	* Evidence that people want it
	* Entities, activities, connections, time
	* The business lifecycle
* Graph is a natural way of modeling data
* Added Events
	* announced at TCDisrupt London
	* built in 2 weeks (!) because of Neo4j. This would have taken months with a RDBMS
* Graph data model adapts easily to changing requirements
* Built in Business Intelligence
	* Don't know what questions to ask in advance (at data modeling time)
	* OLAP alongside graph database, graphDB for ad-hoc queries
* Directly maps to Object Oriented thinking (OGM)
	* Nodes = objects, etc.
* Why doesn't everybody use Neo4j?
	* Historical reasons
		* Databases are used as dumb data stores
		* Logic / analysis is in application layer, focus on logic not data
	* Not plug and play
		* not supported by many high level languages
		* have to think about data modeling
		* CrunchBase developed Deja, a Ruby ORM for Neo4j
* Graph Insights feature on CrunchBase would not be possible without a graph database

## Using Graphs for Next-Gen Master Data Management at Pitney Bowes

### Aaron Wallace (Principal Product Manager, Pitney Bowes)

Slides: http://www.slideshare.net/neo4j/using-graphs-for-nextgen-master-data-management-at-pitney-bowes

* Integrate the data silos
* Single view of individual and connections and interactions (transactions)
* Insights / predictive analytics are now critical for companies
* Visual schema managment
* Visual query builder
* Graph database are peaking on the Gartner "Hype Cycle"
* Pull data from data silos and building the knowledge graph
* Easily expose custom built queries as a web service (REST/SOAP)

## From Zero To Graph In 120: Query

### Nicole White (Data Scientist, Neo Technology)

* Flight data set
* How to use Cypher to query this data set
* For performance, filter by cardinality first
	* Example query went from 1500+ms to 198ms

## From Zero To Graph In 120: Scale

### Ian Robinson (Engineer, Neo Technology)

![](/static/images/notes-from-graphconnect-2014/IMG_1446.png)


Slides: http://www.slideshare.net/neo4j/scaling-neo4jv1

* Understand current needs
* Iterative and incremental development
* Unit testing
* Performance tests

### Scaling Reads - Latency

* Latency is a function of search area
* Search area is a function of of domain invariants
* Domain invariants
	* absolute - (constant query latency x=50)
	* relative - x% of data set
		* query latency increases as data grows
* How to reduce latency?
	* Change domain invariants (often not possible)
	* Improve Cypher query
		* Small queries
		* Separate using **WITH** statements
		* Filter lower cardinality first
	* Change model
		* Explore less of the graph (reduce search area)
		* Using inferred (implicit) relationships?
			* replace inferred relationships with actual relationships (edges in the graph)
			* Cheaper reads, but more expensive write
			* More data to store
			* When to update these? On insertion (same transaction)? Batch updates?
	* Use unmanaged server extensions (custom API)
		* Write Java code that runs in the server
		* Benefits
			* closer to the data
			* single request, many operations
			* multiple implementation options
				* Cypher, Traversal API, Graph Algorithms, Java API
			* control request / response format - compact, conserve bandwidth
			* control HTTP header - cache control

### Scaling Reads - Throughput

* Scale horizontally - cluster
* HAProxy as read load balancer
* Object cache data is evicted
	* Solved with cache sharing (consistent routing)
		* Add parameter in URI for consistent routing, HAProxy config

### Scaling Writes

* Contention for locks
	* Lock nodes at last moment
	* Batch writes
		* Introduce a queue layer. Can add queue inside server extension

## Panel: Visualizing Graphs

### Cambridge Intelligence, Tom Sawyer Software, Linkurious and GraphAlchemist

![](/static/images/notes-from-graphconnect-2014/IMG_1447.png)


* People don't realize graphs are powering stuff (Facebook, recommendations, etc)
* Find one dimension that holds data together (ex: maps and location), but with graphs can have unlimited dimensions. Find one dimension for visualization to make sense.
* Groups / clustering. Show how groups are related, not individuals
* Trend: specific products / implementations for specific use cases
* Alternative to circles and lines?
	* island approach. remove edges
	* reduce number of edges for best visuals
	* matrix representation
* Every aspect of the visualization can be bound to the data (color, width, etc)
* 3D visualizations?
	* technology is not quite ready yet
	* 2.5D shows some promise


## Betting The Company On A Graph Database - Part 2

### Aseem Kishore (Developer, FiftyThree)

Slides available [here](http://aseemk.com/talks/mix-neo4j).

![](/static/images/notes-from-graphconnect-2014/IMG_1448.png)


* Mix by 53
	* Platform for collaborative creation
* Relationship direction choice
	* Point to the thing that the other depends on
* Use linked list in-graph for better performance
* Streams - profile stream
* Pagination
	* Must deal with changing data
	* Instead of skip, use cursor time and filter based on that
* Home stream (with recommendations)
* Deduplication
	* **WHERE NOT** is expensive
	* Use **PROFILE** to profile query plan
* Logarithmic threshold (???)

## Neo4j At Scale Using Enterprise Integration Patterns

### Brad Nussbaum (CTO, MediaHound)

Slides: http://www.slideshare.net/neo4j/media-hound-graphconnectfinal

![](/static/images/notes-from-graphconnect-2014/IMG_1450.png)


* Mediahound is building the entertainment graph
* Collaborative filtering for recommendations
* Graph influencers
* Requirement: sustained user write load and handle batch updates
* Use transactional cypher endpoint
	* Best for batch updates
* Writing a single relationship is 33 byte write to disk
* As of 2.2 Neo4j will batch writes on server

### Enterprise integration patterns

* Splitters - break down larger messages
* Aggregators - batching
* Throttling - limit concurrent writes. Use Neo4j status codes
* Push factor - how many replications must be written to for a successful return
* Cypher is the standard
	* Check driver for transaction support
* **"USING INDEX"** hint
* Recommendation processing with AWS spot instances
* Server extensions
	* Cache abstraction with Google Guava. Large in-memory caching.

### AWS Sport Instance Grid Processing

* Spot instances are bid-based and can be killed at any time if out bid
* Used for concurrent job processing
* Use Cypher statements that are not dependent
* Compound statements for Node/Relationship creation for the subgraph

### ESB

* 400 to 2000 updates per transaction is optimal
* Insert nodes -> single lock
* Insert relationships -> 2 node lock, so do node insertions first

## Applying The GraphAware Framework

### Michal Bachman (Director, GraphAware)

Slides: http://www.slideshare.net/neo4j/graphconnect-2014-sf

@graph_aware

@bachmanm

* GraphAware combines common patterns into a framework
* [Open sourced](http://graphaware.com/)

### Advanced Use Cases

* Custom APIs
* Transaction driven behavior
* Async / background processing

### Custom APIs

* access to Java API for improved performance
* Locking - selective locking
* use-case too complex for Cypher
* Custom logic
* limit to read only
* custom I/O format
* code to data distance
* GraphAware
	* Spring MVC
	* testing with GraphUnit
* Demo
	* Time hierarchy with GA TimeTree
	* `/timetree/now?resolution=second` - creates hierarchy
	* `/timetree/range...` - query based on range


### Transaction Driven Behavior

* ACID
* Integration with other systems
* in-graph indexing
* additional modifications per transaction
* schema enforcement
* Demo
	* Track changes in graph
	* Assign UUIDs

### Async / Background computation

* Crawlers
* Neo4j is very good at OLTP
* Graph algorithms can be expensive in real-time, but many can be approximated
* Execute computation during quiet periods
	* Adapts as usage changes
* PageRank (random walk, incremental counter)
* Recommendation
* Similarities
* Centralities
* Statistics
* Demo
	* Random graph generation
	* run NodeRank in background
* Recommendation system is WIP for GraphAware


## Keynote: Impossible Is Nothing With Graphs

### Jim Webber (Chief Scientist, Neo Technology)


![](/static/images/notes-from-graphconnect-2014/IMG_1456.png)


* Graph databases revolutionize data
* Reliabiliity, availability, eventual consistency
* As Neo4j is becoming more aggressive, how is this impacting the technology?

### History of data

* 70s
	* databases reflected squareness of data
* 80s
	> Databases designed for boring payroll processing, not your geospatial, social, mating mashup.

* 90s
	* Data got big
	* The Web! Graph data at scale, but nobody realized it because they were too busy dealing with app servers and ORMs
	* "The Vietnam decade of computer science"
* 00s
	* Large web properties building their own databases
	* Scale - NoSQL to the rescue!
	* Most NoSQL DBs deal with aggregate (unconnected) data
	* MapReduce to find connections
	* With graph database connections are first class citizens
	* Richer model with connected data

### Graph Database Use-Cases

* Telenor - manage hierarchy of customers
* Beamly - second screen startup. Metadata about TV
* eBay Now - quick routing
	* Before Neo4j, had queries longer than their shortest deliveries
* Accenture - saved Christmas with improvements on parcel delivery routing
* CrunchBase - the Business Graph


* Neo4j is still a small (scrappy) startup


### Reliability vs Availability

* Lee-Anderson - Fault Tolerance
* Dependability
	* Tradeoff between availability (uptime) and reliability (probability to produce correct output)
	* Integrity
	* Safety
	* Confidentiality
* Should the system be available or reliable?
	* depends
	* Aggregate database -> agree on a single value
	* Graph database -> 2 nodes in relationship both have to agree
	* For Graphs: reliability over availability

### What's Next?

* Bigger data (trillions of nodes)
* Smarter Cypher, learn from previous queries
* More throughput
* How is this possible?
	* Native graph all the way down
		* Optimize everywhere

GraphConnect is an annual conference for graph database enthusiasts. These are my notes from GraphConnect 2014 in San Francisco.

Notes From GraphConnect 2014



![](/static/images/boulder-startupweek-2014/IMG_0534.png)

I had the opportunity to spend the last week in Boulder, CO for Boulder StartupWeek. StartupWeek is a week long event highlighting the startup and tech scene of a communty. It started in Boulder, but is now spreading. I wanted to share my experience and any takeways.

*My notes were somewhat incomplete; apologies for all errors and omissions.*

### Monday

#### Development tools forum

The kickoff event of the week was a series of lightening talks about development tools. This event was held in the [SCRiB co-working space](http://scrib.co/):

![](/static/images/boulder-startupweek-2014/IMG_0492.png)

One talk I found particularly interesting was presented by [Nick Quinlan](https://twitter.com/yaynickq) of Sendgrid. He highlighted some useful tools for exploring APIs. Slides are available [here](http://talks.nicholasquinlan.com/2014/05/12/get-api-tools/)

#### Futurist Thinking

A panel talk about futurist thinking and the singularity. Insights on where technology is leading us and impact that is having on society. One panelist had an interesting take on why social media and sharing are so pervasive. He pointed to recent studies that have shown the brain releases oxytocin (a hormone associated with pleasure and bonding) when sharing via social media. His suggestion for entrepreneurs: find a product that causes the release of oxytocin for your customers.

#### Sphero Open House

[Orbotix, Inc](http://gosphero.com), the makers of the Sphero robot were kind enough to open up their offices for an open house. Beer was on tap and robots were rolling around the obstacle course. [Ollie](http://www.gosphero.com/ollie-builder/), the latest robot offering from Orbotix was on display.

![](/static/images/boulder-startupweek-2014/IMG_0499.png)

#### Boulder Beta

The Boulder Beta event involved six Boulder startups demonstrating their products. One that I was particularly excited about was [Ramen](https://ramen.is/), a crowdfunder for software projects. Ramen allows you to fund your next software project with pre-sales, but also to get design feedback. Their product wraps sales/funding, customer validation and user feedback into one service. As anyone who has tried to build a software product knows, these are not easy things to manage. Ramen is live now, I encourage any developers working on building a software product now to check it out.


### Tuesday

#### Functional programming

Despite being scheduled at 8am, the talk on functional programming (led by [Brian](https://twitter.com/puffnfresh) and [Kris](https://twitter.com/nuttycom) of [Simple Energy](http://simpleenergy.com/)) was standing room only. Brian focused on an introduction to the functional programming paradigm, leading with this appropriate quote:

> ... functional programming is a restriction on how we write programs, but not on what programs we can express.

Kris focused on how to write the perfect functional program. They were kind enough to post slides for those interested: [here](http://brianmckenna.org/files/presentations/functional-programming.pdf) and [here](http://nuttycom.github.io/startup_week-2014/#/how-to-write-a-perfect-program).

#### Design Tools

![](/static/images/boulder-startupweek-2014/IMG_0514.png)

[Dave Pitman](https://twitter.com/edavepitman) presented in the new [Galvanize Boulder](http://www.galvanize.it/locations/#boulder-west-pearl) space. In fact, this was the first event to make use of the space since its remodel. Dave made the case that static design mockups are no longer sufficient when today's designs need to reflect complex user interactions like touch gestures and animations. He talked about [five design tools](http://www.slideshare.net/DavidPitman/crafting-awesome-designs-bsw14) that he uses to help bridge this gap.

#### Startup Crawl

Tuesday night concluded with a "startup crawl": eleven startups opened their offices, providing beer and fun events. This was a great opportunity to see what some of Boulder's startup office environments are like, meet lots of startup folks and see what they're working on.

**Bughouse at the Simple Energy Office**

![](/static/images/boulder-startupweek-2014/IMG_0530.png)

**Craps at MobileDay / JumpCloud**

![](/static/images/boulder-startupweek-2014/IMG_0527.png)

**Amazing patio at Galvanize Boulder**

![](/static/images/boulder-startupweek-2014/IMG_0510.png)

**New Sendgrid offices**

![](/static/images/boulder-startupweek-2014/IMG_0487.png)


### Wednesday

#### Big Data Lightning Talks

Wednesday started for me with a series of lightening talks about Big Data and how companies are solving Big Data problems. Usually I cringe when I hear the phrase "big data", but these talks were quite compelling. Two of the three presenters talked about modeling their problem as a graph, something I'm seeing more and more in the big data space. As someone studying network science and dealing with my own big data graph problems I really believe applied graph theory is one of the more useful paradigms for big data solutions.

#### Product Videos

Becca and Larkin from [RiverandLark Productions](http://riverandlark.com/), a video production studio based in Boulder presented about how to create a compelling product video. The takeaway for me was to focus on telling a story and capturing human emotion, as that is the level that makes your story most relateable.

#### Analyze Boulder

![](/static/images/boulder-startupweek-2014/IMG_0546.png)

Analyze Boulder is a meetup group that meets regularly to talk about all things data. Here's my short summary of each of the talks presented:

* **Boulder Bike Share** - Boulder has a great BCycle bike share program. Analyzing the data from how people are using the bike share reveals that people don't like to ride uphill, when it's cold or when it's raining. Great overview of how mining usage patterns to make smart decisions can improve city infrastructure offerings.

* **Algorithmically detecting assholes** - Analyzing game play from Activision's Call of Duty to train a model to automatically find cheaters. Data science is everywhere, even in your video games!

* **Tomnod** - crowdsourced satellite imagery analysis. Where is flight MH-370? Tomnod's platform allowed for a million users to analyze satellite photos in an attempt to find the missing aircraft. How else could this platfrom be used? From search and rescue missions to evaluating oil fields.

### Thursday

#### Walking Tour Of Boulder

![](/static/images/boulder-startupweek-2014/IMG_0553.png)

Frank from [Boulder Walking Tours](http://boulderwalkingtours.com/) led us around downtown Boulder and told us some great stories from Boulder's early days. Did you know that Boulder currently has a 55ft maximum building height? And that is also the height of the tallest tree in Boulder?

#### Techstars For A Half Day

![](/static/images/boulder-startupweek-2014/IMG_0555.png)

It's obvious that Techstars has had a huge impact on the startup community in Boulder. This event gave some insight into what it's like to be a part of the Techstars program. Techstars mentors were available to listen to pitches and give feedback to help hone your startup pitch and business model ideas. I got some great feedback for a project I'm working on and enjoyed hearing what others are working on.

#### Boulder Ignite

![](/static/images/boulder-startupweek-2014/IMG_0562.png)

20 slides. 15 seconds per slide. I heard this event compared to TED talks, but the format really allows for a much wider range of stories to be told in a much more concise form. Inspiring talks about overcoming your fears, traveling adventures, the power of family members, the benefits of the hemp industry and much more.

### Friday

#### Early Stories At Big Companies

Andrew Hyde, lead organizer for StartupWeek, presenting the panel for a talk on early stories at big companies. Panel members were founders and early hires from Sovrn, Twitter, Sendgrid and GitHub.

![](/static/images/boulder-startupweek-2014/IMG_0568.png)

Takeways for me:

* Competition from large companies is not necessarily a bad thing. It validates your idea, can give you better visability, and offers the opportunity to win on implementation (Sendgrid)
* Be wary of PR firms early on. Don't let them convince you have something worth while when you really don't (Sovrn).

#### Hiking With Angels And VCs

![](/static/images/boulder-startupweek-2014/IMG_0573.png)

Boulder has easy access to outdoor recreation opportunities. An example of that is Chautauqua Park. Great hiking trails to explore. How about sharing those trails with a group of entrepreneurs and angel investors?

![](/static/images/boulder-startupweek-2014/IMG_0579.png)


Thanks so much to everyone involved in StartupWeek, especially the organizers and volunteers. And thanks to the sponsors for sponsoring my trip to Boulder as part of the fly-in program. I met so many great people and got to experience a piece of the Boulder startup community.

My experiences during Boulder StartupWeek 2014.

Boulder StartupWeek 2014


![Neo4j Spatial Browser](/static/images/mapping-the-worlds-airports-with-neo4j-spatial-and-openflights-part-1/Screen_Shot_2014_03_13_at_8_53_30_AM.png)

When I was taking flight lessons, I learned that the maximum range of a [Cessna-172](http://en.wikipedia.org/wiki/Cessna_172) is about 1000km. Knowing this, **can we find a path across the globe where each airport in the path is within the range of a Cessna 172?** Let's use Neo4j Spatial to try to answer this question! This blog post will focus on loading data from [OpenFlights.org](http://openflights.org) into Neo4j and how we can use Neo4j Spatial to help us represent this data in a Neo4j graph database. The next post will show how we can use the Neo4j Spatial Java API to traverse our graph and answer this question.

## Neo4j Spatial
[Neo4j Spatial](https://github.com/neo4j/spatial) is a Neo4j Server plugin that allows for spatial operations within a Neo4j database. Spatial makes use of an in-graph [R-tree](http://en.wikipedia.org/wiki/R-tree) to create spatial indexes allowing for geographic queries in the context of Neo4j. For example, Spatial allows for queries to find nodes within a specified geographic region or within a certain distance of a specified point. Neo4j Spatial can easily be installed as a server extension. Pre-packaged ZIPs containing all the JARs needed are provided on the [Spatial GitHub page](https://github.com/neo4j/spatial). Alternatively, [GrapheneDB](http://graphenedb.com) provides Neo4j Spatial as a plugin option. Getting Spatial running on a GrapheneDB instance is literally as easy as a few clicks.

## OpenFlights.org Data
OpenFlights.org provides a [dataset](http://openflights.org/data.html) containing information about almost every airport in the world. This dataset includes 6977 airports, in CSV format:

* Airport ID
* Name
* City
* Country
* **IATA/FAA code**
* ICAO code
* **Latitude**
* **Longitude**
* Altitude
* Timezone
* DST

We are most interested in columns 4 (airport FAA code) and 6/7 (latitude/longitude). Here's what the row for my home airport, MSO, looks like:


```md
4216,"Missoula Intl","Missoula","United States","MSO","KMSO",46.916306,-114.090556,3205,-7,"A"
```

## Loading The Data
Like most Neo4j projects, we have several API choices for how we want to interact with the Neo4j Server. In the next post in this series we'll look at using the Spatial Java API, but for now we will use the REST interface. To make use of Spatial we will follow three steps:

1. Create a Spatial index
1. Create nodes with lat/lon data as properties
1. Add these nodes to the Spatial index

We also will create an index in Neo4j (with the same name as our Spatial layer) to that we can query the database using Cypher.


## Neo4j Spatial REST API
This Python script follows the steps outlined above and uses the REST interface to add each airport to the database.

```python
import csv
import requests
import json

headers = {'content-type': 'application/json'}

# Create geom index
url = "http://localhost:7474/db/data/index/node/"
payload= {
  "name" : "geom",
  "config" : {
    "provider" : "spatial",
    "geometry_type" : "point",
    "lat" : "lat",
    "lon" : "lon"
  }
}
r = requests.post(url, data=json.dumps(payload), headers=headers)

with open('../data/airports.dat', 'rb') as f:   # read data file
    reader = csv.reader(f)
    for row in reader:
        # create airport node
        url = "http://localhost:7474/db/data/node"
        payload = {'lon': float(row[7]), 'lat': float(row[6]), 'name': row[4]}
        r = requests.post(url, data=json.dumps(payload), headers=headers)
        node = r.json()['self']
        
        #add node to geom index
        url = "http://localhost:7474/db/data/index/node/geom"
        payload = {'value': 'dummy', 'key': 'dummy', 'uri': node}
        r = requests.post(url, data=json.dumps(payload), headers=headers)
        
        #add node to Spatial index
        url = "http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/addNodeToLayer"
        payload = {'layer': 'geom', 'node': node}
        r = requests.post(url, data=json.dumps(payload), headers=headers)
```

First, the Spatial index is created, specifing the geometry type and properties that will contain lat/lon data on each node to be added to that index. The file `airports.dat` is our OpenFlights CSV file listing all airports. We iterate through each row, creating a node for each airport, adding that node to the Spatial index and then to the index we have created so we can later query using Cypher.

## Finding Airports

All airports should now be loaded into our database. Now we can start to query the database using some Spatial features to find airports.

### Using Cypher

The following Cypher query will search for an airport within 50km of my hometown, Missoula, MT.

```cypher
// Find airport within 50km of Missoula, MT
START n=node:geom('withinDistance:[46.9163, -114.0905, 50.0]') RETURN n
```

Running that in Neo4j we see:
![MSO query](/static/images/mapping-the-worlds-airports-with-neo4j-spatial-and-openflights-part-1/Screen_Shot_2014_03_13_at_8_29_02_AM.png)

Alright, looks like Missoula has an airport! Great!

### Using the REST interface

We can also make use of the REST interface to find all airports within a certain georgraphic area, or within a certain distance of a specified point.

The endpoint `/db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance` will allow us to query for all airports within 175km of Missoula, MT:

```md
POST http://localhost:7474/db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance
BODY
{
  "layer" : "geom",
  "pointY" : 46.8625,
  "pointX" : -114.0117,
  "distanceInKm" : 175
}
```

```json
[
    {
        "labels": "http://localhost:7474/db/data/node/8358/labels",
        "outgoing_relationships": "http://localhost:7474/db/data/node/8358/relationships/out",
        "data": {
            "lon": -114.090556,
            "bbox": [
                -114.090556,
                46.916306,
                -114.090556,
                46.916306
            ],
            "name": "MSO",
            "lat": 46.916306,
            "gtype": 1
        },
        "traverse": "http://localhost:7474/db/data/node/8358/traverse/{returnType}",
        "all_typed_relationships": "http://localhost:7474/db/data/node/8358/relationships/all/{-list|&|types}",
        "property": "http://localhost:7474/db/data/node/8358/properties/{key}",
        "self": "http://localhost:7474/db/data/node/8358",
        "properties": "http://localhost:7474/db/data/node/8358/properties",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/8358/relationships/out/{-list|&|types}",
        "incoming_relationships": "http://localhost:7474/db/data/node/8358/relationships/in",
        "extensions": {},
        "create_relationship": "http://localhost:7474/db/data/node/8358/relationships",
        "paged_traverse": "http://localhost:7474/db/data/node/8358/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "all_relationships": "http://localhost:7474/db/data/node/8358/relationships/all",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/8358/relationships/in/{-list|&|types}"
    },
    {
        "labels": "http://localhost:7474/db/data/node/7968/labels",
        "outgoing_relationships": "http://localhost:7474/db/data/node/7968/relationships/out",
        "data": {
            "lon": -112.497472,
            "bbox": [
                -112.497472,
                45.954806,
                -112.497472,
                45.954806
            ],
            "name": "BTM",
            "lat": 45.954806,
            "gtype": 1
        },
        "traverse": "http://localhost:7474/db/data/node/7968/traverse/{returnType}",
        "all_typed_relationships": "http://localhost:7474/db/data/node/7968/relationships/all/{-list|&|types}",
        "property": "http://localhost:7474/db/data/node/7968/properties/{key}",
        "self": "http://localhost:7474/db/data/node/7968",
        "properties": "http://localhost:7474/db/data/node/7968/properties",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/7968/relationships/out/{-list|&|types}",
        "incoming_relationships": "http://localhost:7474/db/data/node/7968/relationships/in",
        "extensions": {},
        "create_relationship": "http://localhost:7474/db/data/node/7968/relationships",
        "paged_traverse": "http://localhost:7474/db/data/node/7968/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "all_relationships": "http://localhost:7474/db/data/node/7968/relationships/all",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/7968/relationships/in/{-list|&|types}"
    },
    {
        "labels": "http://localhost:7474/db/data/node/6931/labels",
        "outgoing_relationships": "http://localhost:7474/db/data/node/6931/relationships/out",
        "data": {
            "lon": -111.98275,
            "bbox": [
                -111.98275,
                46.606806,
                -111.98275,
                46.606806
            ],
            "name": "HLN",
            "lat": 46.606806,
            "gtype": 1
        },
        "traverse": "http://localhost:7474/db/data/node/6931/traverse/{returnType}",
        "all_typed_relationships": "http://localhost:7474/db/data/node/6931/relationships/all/{-list|&|types}",
        "property": "http://localhost:7474/db/data/node/6931/properties/{key}",
        "self": "http://localhost:7474/db/data/node/6931",
        "properties": "http://localhost:7474/db/data/node/6931/properties",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/6931/relationships/out/{-list|&|types}",
        "incoming_relationships": "http://localhost:7474/db/data/node/6931/relationships/in",
        "extensions": {},
        "create_relationship": "http://localhost:7474/db/data/node/6931/relationships",
        "paged_traverse": "http://localhost:7474/db/data/node/6931/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "all_relationships": "http://localhost:7474/db/data/node/6931/relationships/all",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/6931/relationships/in/{-list|&|types}"
    },
    {
        "labels": "http://localhost:7474/db/data/node/8169/labels",
        "outgoing_relationships": "http://localhost:7474/db/data/node/8169/relationships/out",
        "data": {
            "lon": -114.256,
            "bbox": [
                -114.256,
                48.310472,
                -114.256,
                48.310472
            ],
            "name": "FCA",
            "lat": 48.310472,
            "gtype": 1
        },
        "traverse": "http://localhost:7474/db/data/node/8169/traverse/{returnType}",
        "all_typed_relationships": "http://localhost:7474/db/data/node/8169/relationships/all/{-list|&|types}",
        "property": "http://localhost:7474/db/data/node/8169/properties/{key}",
        "self": "http://localhost:7474/db/data/node/8169",
        "properties": "http://localhost:7474/db/data/node/8169/properties",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/8169/relationships/out/{-list|&|types}",
        "incoming_relationships": "http://localhost:7474/db/data/node/8169/relationships/in",
        "extensions": {},
        "create_relationship": "http://localhost:7474/db/data/node/8169/relationships",
        "paged_traverse": "http://localhost:7474/db/data/node/8169/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "all_relationships": "http://localhost:7474/db/data/node/8169/relationships/all",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/8169/relationships/in/{-list|&|types}"
    }
]
```

Looking at the response we see that MSO (Missoula, MT), BTM (Butte, MT), HLN (Helena, MT) and FCA (Kalispell, MT) are airports within 175km of Missoula.

## Looking Forward
That was a quick introduction to Neo4j Spatial. The next post will show how we can use the Neo4j Spatial Java API for more advanced traversals and answer our original question: Is it possible to fly a Cessna 172 across the world and what is the route?

Loading every airport in the world into Neo4j Spatial for the purposes of route finding.

Mapping Airports of the World With Neo4j Spatial and Openflights


![Twizzard mobile screenshot](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/Screen_Shot_2014_03_10_at_10_00_10_PM.png)

I spent this past weekend hunkered down in the basement of the local Elk's club, working on a project for a hackathon. The project was a tweet ranking web application. The idea was to build a web app that would allow users to login with their Twitter account and view a modified version of their Twitter timeline that shows them tweets ranked by importance. Spending hours every day scrolling through your timeline to keep up with what's happening in your Twitter network? No more, with [Twizzard](http://www.twizzardapp.com)!

## System structure
Here's a diagram of the system:

![Twizzard system diagram](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/yelp_data_model__3_.png)

* Node.js web application (using Express framework)
* MongoDB database for storing basic user data
* Integration with Twitter API, allowing for Twitter authentication
* Python script for fetching Twitter data from Twitter API
* Neo4j graph database for storing Twitter network data
* Neo4j unmanaged server extension, providing additional REST endpoint for querying / retrieving ranked timelines per user

## Hosted platforms
Being able to spin up hosted instances of our tech stack simplifies the process of putting this project together.

* [GitHub](http://github.com) - It goes without saying that we used GitHub for version control, making collaboration with my [teammate](http://www.visionaryg.com) who handled all the design work quite pleasant.
* [Heroku](http://heroku.com) - Heroku's PaaS for web hosting is great. Free tier is perfect for getting small projects going.
* [MongoLab](http://mongolab.com) - This was my first time using MongoLab's hosted MongoDB service. No complaints, worked great! Also free tier was perfect for getting started.
* [GrapheneDB](http://grapheneDB.com) - GrapheneDB provides hosted Neo4j instances. These guys are awesome! Their service is rock solid. I can't overstate how impressed I am with what they provide (they even allow for running custom server extensions!)

## Getting started
I stumbled acros this node.js [hackathon starter template](https://github.com/sahat/hackathon-starter) a few months ago. I decided to give it a try for the first time this weekend. It puts together a stack that I'm familiar with: node.js, mongoDB, Express, passport for handling OAuth, Bootstrap and jade. It's a great starter template for, as the name implies, starting hackathon projects.


## Graph data model
Since this project deals with Twitter data, modeling that data as a graph seems intuitive. We're concerned with users, their tweets, and the interactions between users. The data model is pretty simple:

![Twizzard graph data model](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/ok__1_.png)

## Inserting Twitter Data With py2neo
Once a user authenticates to our web application and grants us permission to access their Twitter data, we need to access the Twitter API and store the data in Neo4j. We accomplish this with the help of the Python [py2neo](https://github.com/nigelsmall/py2neo) package.

```python
def cypher_insert(friends, timeline, retweets, ME):    
  create_user_query = '''MERGE(u:User {id: {me} })'''
  
  create_friend_query = '''MATCH (me:User {id: {me} })
    MERGE (u:User {id: {id} })
    MERGE (me)-[:FOLLOWS]->(u)'''
  
  add_tweet_query = '''MATCH (u:User {id: {user_id_str}})
    MERGE (t:Tweet {id: {id_str}, image: {image}, 
    user_name: {screen_name}, text: {text}, time: {time}})
    MERGE (t)<-[:TWEETED {created_at: {time}, time: {id_str}}]-(u)'''
  
  add_retweet_query = '''MATCH (u:User {id: {user_id_str}})
    MERGE (t:Tweet {id: {id_str}, image: {image}, 
    user_name: {screen_name}, text: {text}, time: {time}})
    MERGE (t)<-[:RETWEETED {created_at: {time}, time: {id_str}}]-(u)'''
  
  # create_user_query
  create_me_batch = WriteBatch(graph_db)
  create_me_batch.append_cypher(create_user_query, ME)
  create_me_batch.run()
  
  print "Beginning user batch"
  user_batch = WriteBatch(graph_db)
  for user_id in friends['ids']:
      user = {}
      user['id'] = str(user_id)
      user['me'] = ME['me']
      print user
      user_batch.append_cypher(create_friend_query, user)
  user_batch.run()
  
  print "Beginning tweet batch"
  tweet_batch = WriteBatch(graph_db)
  for tweet in timeline:
      tweet_batch.append_cypher(add_tweet_query, tweet)
  tweet_batch.run()
  
  print "Beginning retweet batch"
  retweet_batch = WriteBatch(graph_db)
  for tweet in retweets:
      retweet_batch.append_cypher(add_retweet_query, tweet)
  retweet_batch.run()
```

## Ranking tweets
How can we score Tweets to show users their most important Tweets? Users are more likely to be interested in tweets from users they are more similar to and from users they interact with the most. We can calculate metrics to represent these relationships between users, adding an inverse time decay function to ensure that the content at the top of their timeline stays fresh.

### Jaccard similarity index
The [Jaccard index](http://en.wikipedia.org/wiki/Jaccard_index) allows us to measure similarity between a pair of users. For our purposes this is defined as the intersection of their sets of followers divided by the union of their sets of followers. This results in a score between 0 and 1 representing how "similar" the two users are to each other.

![](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/mathjax.png)

Calculating this in Neo4j Cypher for all users in our database looks like this:

```cypher
// Jaccard index
MATCH (u1:User), (u2:User) WHERE u1 <> u2
MATCH (u1)-[:FOLLOWS]->(mutual)<-[:FOLLOWS]-(u2) WITH u1, u2, count(mutual) as intersect
MATCH (u1)-[:FOLLOWS]->(u1_f) WITH u1, u2, intersect, collect(DISTINCT u1_f) AS coll1
MATCH (u2)-[:FOLLOWS]->(u2_f) WITH u1, u2, collect(DISTINCT u2_f) AS coll2, coll1, intersect
WITH u1, u2, intersect, coll1, coll2, length(coll1 + filter(x IN coll2 WHERE NOT x IN coll1)) AS union
```

### Interaction metric

Another important factor to take into account is how often Twitter users are interacting with each other. Users are more likely to be interested in tweets from users they interact with often. To quantify the strength of this relationship for a user pair `A,B`, we simply divide the number of `A,B` Twitter interactions by the total number of interactions for user `A`.

### Weighted average
The similarity score and interaction score are combined using a weighted average. We weight similarity slightly higher than interaction.

### Time decay
To ensure temporal relevence, an inverse time decay function is used to discount tweets according to the amount of time elapsed since the tweet was sent:

![](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/tweetscore.png)


By storing these data and relationships in our Neo4j instance, we simply need to query the database for the highest ranked tweets from our web application and display these to the user.


## Extending the Neo4j REST interface

Neo4j Server provides a REST interface that allows for querying of the database. Neo4j also allows for adding unmanaged server extensions that allow us to extend the built-in REST API and add our own endpoints. We can do this using JAX-RS in Java. In this case, we write a simple server-extension that adds the endpoint `/v1/timeline/{user_id}` that will execute a Cypher query to return the tweets for the specified user's timeline, ordered by the ranking metric we've defined above.

```java
@Path("/timeline")
    public class App 
    {
        private final ExecutionEngine executionEngine;
        public App(@Context GraphDatabaseService database) {
            this.executionEngine = new ExecutionEngine(database);
        }
    
        @GET
        @Produces(MediaType.APPLICATION_JSON)
        @Path("/{user_id}")
        public Response getTimeline(@PathParam("user_id") String user_id){
    		ArrayList<Object> tweets = new ArrayList<Object>();
    		Map<String, Object> params = new HashMap<String, Object>();
   			params.put("user_id", user_id);
            String query = "MATCH (u:User {id: {user_id}})\n" +
                    "MATCH (u)-[s:SIMILAR_TO]->(friend)\n" +
                    "MATCH (friend)-[:TWEETED]->(tweet) RETURN tweet.time AS created_at, tweet.id AS id_str,\n" + 
                     "tweet.text as text, tweet.user_name as name, tweet.image as profile_image_url,\n" + 
                     "tweet.user_name as screen_name, s.coef as score ORDER BY score DESC";
    
            Iterator<Map<String, Object>> result = executionEngine.execute(query, params).iterator();
            while (result.hasNext()) {
                Map<String, Object> row = result.next();
                Map<String, Object> user = new HashMap<String, Object>();
                Map<String, Object> tweet = new HashMap<String, Object>();
    
                user.put("name", row.get("name"));
                user.put("screen_name", row.get("screen_name"));
                user.put("profile_image_url", row.get("profile_image_url"));
    
                tweet.put("created_at", row.get("created_at"));
                tweet.put("id_str", row.get("id_str"));
                tweet.put("text", row.get("text"));
                tweet.put("promoted", false);
                tweet.put("user", user);
                tweets.add(tweet);
            }
    
            Gson gson = new Gson();
            String json = gson.toJson(tweets);
            return Response.ok(json, MediaType.APPLICATION_JSON).build();
        }
    }
```

## Twizzard
We had the site up and running by the time final presentations were scheduled. In fact, we even had time for a few iterations based on user feedback. We called our web app Twizzard, as in Your Twitter Wizard (but with two z's, like a Tweet Blizzard). It's running now at [twizzardapp.com](http://twizzardapp.com). Sign in with your Twitter account and check it out.

![Twizzard screenshot](/static/images/twizzard-a-tweet-recommender-system-using-neo4j/Screen_Shot_2014_03_10_at_8_34_39_PM-1.png)

Shout out to my Twizzard teammates [@kevshoe](http://twitter.com/kevshoe) and [@VisionaryG](http://twitter.com/visionaryg). It was great fun working with you guys at Startup Weekend Missoula 2014!


A system for ranking tweets based on user affinity and time decay.

Twizzard, A Tweet Recommender System Using Neo4j


![Fatality heatmap of inter-state disputes](/static/images/visualizing-correlates-of-war-date-with-leaftlet-js/Screen_Shot_2014_02_11_at_6_35_53_PM.png)

The [Correlates of War](http://www.correlatesofwar.org/) project maintains several data sets that document conflicts throughout history. This includes data on armed conflicts such as inter-state disputes as well as ancillary data that might be correlated with disputes, such as trade, alliances and measures of countries' resources. This data makes for interesting geographic visualizations. This post highlights how [Leaftlet.js](http://leafletjs.com/) can be used to create geographic data visualizations using this data.

### Using Crosslet.js to compare battle related fatalities with national capability

Can we see any relationship between the fatalities a country suffers during disputes and it's national capabilitiy?

[![National Material Capabability vs wartime fatalities](/static/images/visualizing-correlates-of-war-date-with-leaftlet-js/nmc_crosslet.png)](http://battlemapz.herokuapp.com/#vis_2)


#### The Data
This first visual uses two datasets from Correlates of War:

* [National Material Capabilities Index](http://www.correlatesofwar.org/COW2%20Data/Capabilities/nmc4.htm)
> The National Material Capabilites data set contains annual values for total population, urban population, iron and steel production, energy consumption, military personnel, and military expenditure of all state members, currently from 1816-2007. The widely-used Composite Index of National Capability (CINC) index is based on these six variables and included in the data set.

* [the New COW War Data, 1816-2007 v4.0](http://www.correlatesofwar.org/COW2%20Data/WarData_NEW/WarList_NEW.html)
> This data set records all instances of when one state threatened, displayed, or used force against another.

Specifically, three values are computed for each country listed in the dataset:

1. CINC time average - the mean Composite Index of National Capability value over time
2. Average fatality - the mean number of battle-related fatalities per militarized interstate dispute
3. Average fatality per military personnel - average fatality relative to the size of that country's military, measured by the number of military personnel for the observation year.

#### Crosslet.js

An awesome javascript library called [Crosslet](http://sztanko.github.io/crosslet/) that uses [D3](http://d3js.org), [Leaflet.js](http://leafletjs.com) and [Crossfilter.js](http://square.github.io/crossfilter/) to create great data centric geographical visualizations is leveraged here. From the developer's website:

>Crosslet is a free small (22k without dependencies) JavaScript widget for interactive visualisation and analysis of geostatistical datasets. You can also use it for visualizing and comparing multivariate datasets.

Using Crosslet is very straight forward. First build the `config` object that specifys our Leaflet API key and the geoJSON file we wish to use to define the borders of the countries we wish to highlight:

```js
var config = {
  map: {
    leaflet: {
      key: "YOUR_LEAFTLET_KEY_HERE",
      styleId: 64657,
      attribution: "BattleMapz"
    },
    view: {
      center: [35.0, 0.1],
      zoom: 2
    },
    geo: {
      url: "data/world.topo.json",
      name_field: "NAME",
      id_field: "NAME",
      topo_object: "world"
    }
  }
```

Next, we specify which data we want to load.

```js
dimensions: {
    cinc: {
      title: "Composite Index of National Capability (x1000) - time average",
      data: {
        //colorscale: d3.scale.linear().domain([-1500, 5, 200]).range([ "red", "yellow","green"]).interpolate(d3.cie.interpolateLab),
        dataSet: "nmc_data.tsv",
        field: "CINC",
	method: d3.tsv,
	ticks: [0,1,5,30,150],
	exponent: 0.1

      },
      format: {
        input: function() {return Math.round},

     }
      
    },
```

Finally, instantiate a new Crosslet map object using the `config` object we just created, passing the DOM element where it should be rendered on the page:

```js
new crosslet.MapView($("#map"),config);

```

In the Crosslet widget above, you can see that we have three dimensions of data displayed in a [choropleth map](http://en.wikipedia.org/wiki/Choropleth_map). We can also see the distribution of each variable as displayed in a histogram/color map for each variable. One of the powerful features of Crosslet is that we can select a range over which to slice the observations. This piece is powered by [Crossfilter.js](http://square.github.io/crossfilter/). This slicing of the data allows for some interesting visual analysis. By selecting the upper range of the CINC index data do we see a change in the distribution of average fatalities?

#### Scatter plot

This map based visualization makes for interesting exploratory data analysis, but we can use more conventional data visualization techniques to test our hypothesis as well. For example, this scatter plot shows a time-average of the CINC score vs. average battle related fatalities relative to the size of a country's military on a [log-log scale](http://en.wikipedia.org/wiki/Log-log_plot). A regression shows that there is a weak negative relationship between average fatalities per battle and a country's CINC score.

[![National Material Capabability vs wartime fatalities](/static/images/visualizing-correlates-of-war-date-with-leaftlet-js/nmc_scatter.png)](http://battlemapz.herokuapp.com/#vis_4)

This visual was created with the [NVD3 library](http://nvd3.org/), which is built on top of [D3.js](http://d3js.org/).

<hr />

### Visualizing battle related fatalities geographically

Next, visualizing inter-state dispute fatalities throughout history using a heatmap. Can we identify "hot" areas where more fatalities occured?

#### The Data

This visual makes use of [Militarized Interstate Disputes v4.0](http://www.correlatesofwar.org/COW2%20Data/MIDs/MID40.html) and the [Militarized Interstate Dispute Location](http://www.correlatesofwar.org/COW2%20Data/MIDLOC/MIDLOC_v1.1.html) datasets. This data is initially in three csv files. I used a Python script to read in the data and do some initial descriptive data analysis, exporting the data into JSON format:

```md
...
	{
        "dispNum": 3,
        "stDay": 2,
        "stMon": 5,
        "stYear": 1913,
        "endDay": 25,
        "endMon": 10,
        "endYear": 1913,
        "outcome": 4,
        "settle": 3,
        "fatality": 0,
        "fatalPre": 0,
        "maxDur": 177,
        "minDur": 177,
        "hiAct": 8,
        "hostLev": 3,
        "recip": 0,
        "numA": 1,
        "numB": 1,
        "link1": "0",
        "link2": "0",
        "link3": "0",
        "ongo2001": 0,
        "version": 3.1
    }
    ...
```

#### Fatality Heatmap With Heatcanvas

This visual uses the [heatcanvas](https://github.com/sunng87/heatcanvas) library to generate a heatmap overlay to highlight battle related fatality levels throughout the world. The value of each pixel displayed on the map is defined by its distance to points that holds data (in this case fatality level for a dispute).

#### Markercluster

Marker annotation functionality is provided by Leaflet, however the easy to use [Leaftlet.markerCluster](https://github.com/Leaflet/Leaflet.markercluster) plug-in handles overlapping annotations by grouping them and tracing the area of the map covered by a given cluster of annotations.

#### Client JS

Data munging is a bit more involved for this visual. To make this process a bit more syntactically simpler we can use the [sift.js](https://github.com/crcn/sift.js) library. This javascript library provides [MongoDB](http://www.mongodb.org/) like syntax for filtering javascript arrays. It allows us to do things like this:


```js
var sifted = sift({ $in:['foo','bar'] }, ['bob','loblaw','foo']); // sifted = ['foo']

```

This will make things a bit easier for us as we:
1) Join the data sets
2) Create the Leaflet.js Tile Layer and init the Marker Layer and HeatCanvas Tile Layer objects
3) Build the marker text, and finally
4) Add the Layers to the map.

<script src="https://gist.github.com/johnymontana/f450fb192c5c18217be2.js"></script>

```js
var testData={data:[]};
for (var i=1; i<MIDA.length; i++) {
	var midlocCurrent = sift({'dispnum': MIDA[i].dispNum}, battleData)[0];
  var lat = midlocCurrent.latitude;
  var lon = midlocCurrent.longitude;
  var outcome = MIDA[i].outcome;
  var value = MIDA[i].fatality;
  var sideA = sift({'dispNum': midlocCurrent.dispnum, 'sideA': true}, MIDB);
  var sideB = sift({'dispNum': midlocCurrent.dispnum, 'sideA': false}, MIDB);
  if (value != -9){
    testData.data.push({'dispNum': midlocCurrent.dispnum, 'lat': lat, 'lon': lon, 'value': value,'location': midlocCurrent.mid21location,'outcome': outcome,'sideA': sideA,'sideB': sideB,'year': midlocCurrent.year});
    //console.log(lat, lon, value);
  }
}
```

```js
var baseLayer = L.tileLayer('http://{s}.xxx.{z}/{x}/{y}.png', { attribution: 'BattleMapz', minZoom: 2, maxZoom: 18});
var markerLayers = new L.MarkerClusterGroup();
var heatmap = new L.TileLayer.HeatCanvas({},{'step':0.3, 'degree': HeatCanvas.LINEAR, 'opacity': 0.7});
view raw
```

```js
for (var i=0; i<testData.data.length; i++){
  var marker = new L.Marker(new L.LatLng(testData.data[i].lat, testData.data[i].lon));
  var sideANames = '';
  var sideBNames = '';
  for (var a=0; a<testData.data[i].sideA.length; a++){
	  sideANames += testData.data[i].sideA[a].stAbb + ', ';
  }
  for (var b=0; b<testData.data[i].sideB.length; b++){
	  sideBNames += testData.data[i].sideB[b].stAbb + ', ';
  }
      // TODO: map codes to fulltext (countries, outcomes, fatalities)
  marker.bindPopup('Dispute: ' + testData.data[i].dispNum + '&ltbr&gtLocation: ' + testData.data[i].location +'&ltbr&gtFatalities: '+ fatalityDict[testData.data[i].value]+'&ltbr&gtStart year: ' + testData.data[i].year.toString() + '&ltbr&gtSideA: ' + sideANames + '&ltbr&gtSideB: ' + sideBNames + '&ltbr&gtOutcome: ' + outcomeDict[testData.data[i].outcome]);
  markerLayers.addLayer(marker);
  heatmap.pushData(testData.data[i].lat, testData.data[i].lon, testData.data[i].value);
}
```

```js
var overlayMaps = {'Dispute Markers': markerLayers, 'Fatality Heatmap': heatmap};
var controls = L.control.layers(null, overlayMaps, {collapsed: false, autoZIndex: true});
var map = new L.Map('map', {center: new L.LatLng(51.505, -0.09), zoom: 2, layers: [baseLayer]});
controls.addTo(map);
```

Here's how it looks (images link to the actual interactive widget - will take a few seconds to load):

**Militarized inter-state disputes marker cluster**

[![Militarized inter-state disputes marker cluster](/static/images/visualizing-correlates-of-war-date-with-leaftlet-js/Screen_Shot_2014_02_11_at_6_35_17_PM.png)](http://battlemapz.herokuapp.com/#vis_1)

**Fatality heatmap of inter-state disputes**

[![Fatality heatmap of inter-state disputes](/static/images/visualizing-correlates-of-war-date-with-leaftlet-js/Screen_Shot_2014_02_11_at_6_35_53_PM.png)](http://battlemapz.herokuapp.com/#vis_1)


#### Problems with this visual

* The maximum fatality level is 999+, swamping disputes that resulted in thousands, hundreds of thousands, and even millions of fatalities.
* Location is set where the dispute began, not where individual battles occured
* Only 1 datapoint per dispute
* Loading performance is poor, mostly due to the data munging being done in the client. The data should be prepared in the format intended to be used ahead of time.

Thanks to the maintainers of the Correlates of War project. Building these visuals was a sobering experience for me. The full code for this project is available on [github](https://github.com/johnymontana/BattleMapz).


Using data about global militarized interstate disputes (wars) we build some geospatial visualizations to analyze data about war fatalities throughout history.