API Documentation

Using the API

Exversion's API allows you to access any dataset through HTTP Requests. Data the user owns or collaborates on can also be manipulated programmatically, bypassing the web interface completely. In cases where the data is poorly formatted or larger than what the uploader supports, the API may be more convenient

Currently the API returns json or XML for queries, full XML support is coming soon.

Authentication

Exversion makes use of both simple key authentication and OAuth. All actions that control a specific account require a valid OAuth access token. Querying the data, searching, or pulling metadata require only a valid api key.

Finding/Changing Your API Key

Your API key is located under the YOUR ACCOUNT section of your user profile or automatically appended to the sample query url on every dataset page. This key can be changed at any time by clicking the RESET button in the YOUR ACCOUNT section. Resetting your key will not effect your account balance or status.

Generating an OAuth Access Token

If you want to create a dataset, add data to an existing set, or edit individual points of data within a set you must first generate an access token to the account that owns the dataset. These tokens do not expire and should be stored for continual use, but they can be revoked by the user who granted them at any time.

There are two ways to generate Access Tokens. If you need to access multiple accounts you must first create an app on Exversion. Apps are listed in the YOUR APPS section of your user profile and can be added, edited and deleted from there.

Alternatively, if you only want access to your own account, you can use Exversion's built in OAuth Client to generate a valid access token.

Once created each app has a unique id and secret that are used in generating access tokens. First the user must grant authorization to the app. This can be done by directing users to the following url:

http://www.exversion.com/oauth/v2/auth?client_id=[CLIENT ID HERE]&response_type=code&redirect_uri=[REDIRECT URL]

If the client id matches a valid Exversion App the user will be redirected to a page asking them to allow or deny your app. Once they click ALLOW they are redirected back to your app via the redirect url specified when you registered your app.

A code will be appended to the redirect url like so:

http://www.example.com/?code=YTQyNTRjN2Y5MmVlYmE1NWQ1NmUwZDNmMGVkYzk4NzFlNzNhYmFhZDExMmM1OWExY2E2ZWY4N2ZlZDYzYjZlOQ

Once you have the code, you can generate the access_token by sending the following HTTP request back to Exversion

http://www.exversion.com/oauth/v2/token?client_id=[CLIENT ID HERE]&client_secret=[CLIENT SECRET]&grant_type=authorization_code&redirect_uri=[REDIRECT URL]&code=[CODE HERE]

Exversion will respond with the access token in json

{

access_token": "8315796acc79f6a1bfb4e4935aea01362d59ecce",

"expires_in": 3600,

"token_type": "bearer",

"scope": null,

"refresh_token": "da359ceafe501fd2445df0a6c406953264e54c47"

}

Include the access token as a parameter of all OAuth requests to manage data on the user's account.

Data Management

With a valid OAuth access token apps can manage data on Exversion programmatically. All requests require a json HTTP POST request.

POST /dataset/create

Creates a new dataset from parameters supplied. Org should be the id number of the relevant organization (if applicable). Private should be either 1 to keep the dataset private or 0 to leave it public. All other parameters are strings following the same conventions of the upload form on Exversion.

Example:

api/v1/dataset/create
{"access_token":"YmQ1...", "name":"api create", "description":"this was created with the api","source_url":"http://www.exversion.com", "org":0,"source_author":"Me","source_date":"August 7, 2013", "source_contact":"info@exversion.com", "private":0}

Response:

{"status":200,"message":"Success","body":[{"dataset":"EV5XF25RH3MIMPP","sourceURL":"http:\/\/www.exversion.com","sourceAuthor":"Me","sourceDate":"August 7, 2013","sourceContact":"info@exversion.com","uploadedBy":{"id":"1","name":"bellmar"},"description":"this was created with the api","heritage":[],"size":0,"forkchanges":null,"columns":[""]}]}

POST /dataset/push

Adds new data to an existing dataset. Controls for duplicate information by remapping rows of existing identical information.

Missing Values In order to ensure the best possible query results, Exversion normalizes pushed data to conform to the dataset's existing schema. If the dataset is empty a schema is created from the first row submitted through the API. For that reason please make sure there are no missing values (empty strings are acceptable). The API will reject any new data that contains parameters not listed in the schema.

Example:

api/v1/dataset/push
{"access_token":"YmQ1....", "dataset":"[DATASET ID]","data":[{"test":"1","gender":"f","Age":"30","name":"Marianne"}, {"test":"0","gender":"m","name":"Jacek"}]}

Response:

{"status":200,"message":"Success","body":{"inserted":[{"test":"1","gender":"f","age":"30","name":"Marianne"},{"test":"0","gender":"m","name":"Jacek","age":""}],"updated":[],"failed":[]}}

POST /dataset/edit

Edit specific data points. Requires the _id of the specific data point (which is part of the information returned by any dataset query). This is intended to fixes errors, mistakes, corruption or inconsistencies in the data, not to update existing data points with new information. All changes are tracked and can be reverted through the web interface.

Example:

api/v1/dataset/edit
{"access_token":"YmQ1...", "edits":[{"dataset":"[DATASET ID]", "_id":"[DATAPOINT ID]", "changes":{"gender":"f","name":"Tal","age":25}}]}

Response:

{"status":200,"message":"Success","body":[{"test":"0","gender":"f","name":"Tal","age":25}]}

Choosing a Format

By default the API will return JSON, but queries can return other formats if you specify that in the url.

Simple Queries:

https://exversion.com/api/v1/dataset/xml/BVVYOK9DUITSH36?key=xxx

https://exversion.com/api/v1/dataset/json/BVVYOK9DUITSH36?key=xxx

Complex Queries:

https://exversion.com/api/v1/dataset/xml

https://exversion.com/api/v1/dataset/json

Simple Queries

To perform basic queries like retriving data from one dataset with a minimal amount of filtering, access a dataset's metadata or search exversion for data, a simple GET Request is usually sufficient.

GET /dataset/:dataset

Runs a query on the selected dataset and returns the results. Consult the dataset's listing for information on columns.

Example:

https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?key=[YOUR KEY]&lobbyist_first_name=James

Response:

{"status":200,
"message":"Success",
"body":[
	{"_id":"FSBEQAKJQJH059881.299319683364711375720224963","lobbyist_first_name":"James","lobbyist_last_name":"Connolly","filing_year":"2012"},
	{"_id":"FSBEQAKJQJH059857.653845592045361375720225585","lobbyist_first_name":"James","lobbyist_last_name":"Gilliam","filing_year":"2012"},
	{"_id":"FSBEQAKJQJH05984.6746827350097781375720228202","lobbyist_first_name":"James","lobbyist_last_name":"Alexander","filing_year":"2012"},
	{"_id":"FSBEQAKJQJH059878.41251410109141375720234876","lobbyist_first_name":"James","lobbyist_last_name":"Banks","filing_year":"2012"},
	{"_id":"FSBEQAKJQJH059890.543216294539121375720235255","lobbyist_first_name":"James","lobbyist_last_name":"Darnell","filing_year":"2012"},
	{"_id":"FSBEQAKJQJH059884.74339911080541375720244224","lobbyist_first_name":"James","lobbyist_last_name":"Henn","filing_year":"2012"},
]}

Pagination and Limits

By default Exversion limits all queries to 250 records. Depending on the size of each individual row of data this might be too many or too few rows per request. Therefore the exact number of rows returned can be changed with the _limit parameter

https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?[YOUR KEY]&lobbyist_first_name=James&_limit=5

Alternatively, you can get results beyond the first 250 with the _page parameter.

https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?[YOUR KEY]&lobbyist_first_name=James&_page=2

Ranges

You can return all data that fits within a numerical range by submitting a max and min value within square brackets

https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?key=xxx&age=[30:35]

To return only values that are greater than or only those that are less than a certain number, just leave off one end of the range in your query

https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?key=xxx&age=[:35]
https://exversion.com/api/v1/dataset/FSBEQAKJQJH0598?key=xxx&age=[30:]

GET /metadata/:dataset

Returns the metadata about a selected dataset, including ids for the parent datasets if applicable.

Example:

api/v1/metadata/A9G8JOU2O95IPY2?key=xxx

Response:

{"status":200,
"message":"Success",
"body":[
	{"dataset":"A9G8JOU2O95IPY2",
	"sourceURL":"https:\/\/data.sfgov.org\/Arts-Culture-and-Recreation-\/Film-Locations-in-San-Francisco\/yitu-d5am",
	"sourceAuthor":"Lauren Machado",
	"sourceDate":"Nov 10, 2011",
	"sourceContact":"N\/A",
	"uploadedBy":{"id":17,"name":"bellmar"},
	"description":"If you love movies, and you love San Francisco, you're bound to love this -- a listing of filming locations of movies shot in San Francisco from 1924 - 2010. You'll find the titles, locations, fun facts, names of the director, writer, actors, and studio for most of these films.",
	"heritage":[],
	"size":193208,
	"forkchanges":null}
	]
}

GET /search

Search for data through the API. No API Key is required for this call.

Example:

api/v1/search?q=film

Response:

{"status":200,
"message":"Success",
"body":[
	{"dataset":"A9G8JOU2O95IPY2",
	"sourceURL":"https:\/\/data.sfgov.org\/Arts-Culture-and-Recreation-\/Film-Locations-in-San-Francisco\/yitu-d5am",
	"sourceAuthor":"Lauren Machado",
	"sourceDate":"Nov 10, 2011",
	"sourceContact":"N\/A",
	"uploadedBy":{"id":17,"name":"bellmar"},
	"description":"If you love movies, and you love San Francisco, you're bound to love this -- a listing of filming locations of movies shot in San Francisco from 1924 - 2010. You'll find the titles, locations, fun facts, names of the director, writer, actors, and studio for most of these films.",
	"heritage":[],
	"size":193208,
	"forkchanges":null}
	]
	}

Complex Queries

POST /dataset

To run a complex query, POST a well formatted json string to the dataset endpoint. Queries are filtered by adding a column => value pair to the "params" object. For example:

{"key":"[YOUR KEY]","merge":0, "query":[{"dataset":"MXZHQGZSVH8484K", "params":{"state":"CA"}, "_limit":3}]}

Is equivilant to the following simple query:

https://exversion.com/api/v1/dataset/MXZHQGZSVH8484K?[YOUR KEY]&state=CA&_limit=3

Like simple queries, you can paginate through the results or adjust the default limit with "_page" and "_limit" parameters respectively.

To query a range make sure the range block is posted as a string:

{"key":"[YOUR KEY]","merge":0, "query":[{"dataset":"MXZHQGZSVH8484K", "params":{"population":"[1000:10000]"}, "_limit":3}]}

Querying within a column

Some datasets may have subcategories, or columns within columns. You can query by parameters WITHIN a column by altering the params key to reflect the appropriate hierarchy. For example:

{"key":"[YOUR KEY]","query":[{"dataset":"UXHD1UH4JCZ8KHC", "params":{"properties.CNTRY_NAME":"Aruba"}}]}

...will return all the rows where CNTRY_NAME is "Aruba" in the properties column. Please note that keys in this query are CASE SENSITIVE. PROPERTIES.CNTRY_NAME is not the same as properties.cntry_name or properties.CNTRY_NAME.

Calling multiple datasets at once EXPERIMENTAL

You can access multiple datasets with one query and either merge them together or keep them seperated by dataset id. To fully merge multiple datasets together and form a giant array of results, set the merge value of the json POST request to 1. Otherwise the API will assume you want to sort results by dataset id.

Example:

api/v1/dataset
{"key":"[YOUR API KEY]","merge":0, "query":[{"dataset":"NOKZCNE77CX711263", "params":{"zipcode":10454}, "_limit":3, "_page":3},{"dataset":"MXZHQGZSVH8484K", "params":{"state":"CA"}, "_limit":3}]}

Response:

FULL MERGE
{"status":200,
"message":"Success",
"body":[{"_id":"NOKZCNE77CX711263.295905402291431375046116293","test":"0","gender":"m","age":"18","name":"Morris"},{"_id":"NOKZCNE77CX711291.844049101067431375046116294","test":"0","gender":"m","age":"20","name":"Frank"},{"_id":"NOKZCNE77CX711232.961505202183471375046116295","test":"1","gender":"m","age":"33","name":"Larry"},{"_id":"MXZHQGZSVH8484K14.621527933928611375311400627","aggravated_assault":"657","arson2":"51","state":"CALIFORNIA","burglary":"1,741","forcible_rape":"29","robbery":"372","property_crime":"4,757","motor_vehicle_theft":"1,096","year":"2012","city":"ANTIOCH","murder":"10","population1":"105,009","violent_crime":"1,068","larcenytheft":"1,920"},{"_id":"MXZHQGZSVH8484K64.12494301196191375311400673","aggravated_assault":"374","arson2":"22","state":"CALIFORNIA","burglary":"792","forcible_rape":"58","robbery":"181","property_crime":"3,849","motor_vehicle_theft":"353","year":"2011","city":"OCEANSIDE","murder":"7","population1":"","violent_crime":"620","larcenytheft":"2,704"},{"_id":"MXZHQGZSVH8484K33.314552619220921375311400712","aggravated_assault":"191","arson2":"19","state":"CALIFORNIA","burglary":"1,206","forcible_rape":"35","robbery":"156","property_crime":"3,952","motor_vehicle_theft":"443","year":"2011","city":"FREMONT","murder":"2","population1":"","violent_crime":"384","larcenytheft":"2,303"}]}
NO MERGE
{"status":200,
"message":"Success",
"body":{
	"NOKZCNE77CX7112":[{"_id":"NOKZCNE77CX711263.295905402291431375046116293","test":"0","gender":"m","age":"18","name":"Morris"},{"_id":"NOKZCNE77CX711291.844049101067431375046116294","test":"0","gender":"m","age":"20","name":"Frank"},{"_id":"NOKZCNE77CX711232.961505202183471375046116295","test":"1","gender":"m","age":"33","name":"Larry"}],

	"MXZHQGZSVH8484K":[{"_id":"MXZHQGZSVH8484K14.621527933928611375311400627","aggravated_assault":"657","arson2":"51","state":"CALIFORNIA","burglary":"1,741","forcible_rape":"29","robbery":"372","property_crime":"4,757","motor_vehicle_theft":"1,096","year":"2012","city":"ANTIOCH","murder":"10","population1":"105,009","violent_crime":"1,068","larcenytheft":"1,920"},{"_id":"MXZHQGZSVH8484K64.12494301196191375311400673","aggravated_assault":"374","arson2":"22","state":"CALIFORNIA","burglary":"792","forcible_rape":"58","robbery":"181","property_crime":"3,849","motor_vehicle_theft":"353","year":"2011","city":"OCEANSIDE","murder":"7","population1":"","violent_crime":"620","larcenytheft":"2,704"},{"_id":"MXZHQGZSVH8484K33.314552619220921375311400712","aggravated_assault":"191","arson2":"19","state":"CALIFORNIA","burglary":"1,206","forcible_rape":"35","robbery":"156","property_crime":"3,952","motor_vehicle_theft":"443","year":"2011","city":"FREMONT","murder":"2","population1":"","violent_crime":"384","larcenytheft":"2,303"}]}}

Joining Dataset Together Across a Column HIGHLY EXPERIMENTAL

In situations where multiple datasets follow of relational model, the API can join them across a single column so that data from each set is returned in each row.

Request is set up like a normal query, except that you include the name of the columns that where the dataset and the joinset are to be joined.

Example:

api/v1/dataset
{"key":"fac59c51c0","query":[{"dataset":"MXZHQGZSVH8484K", "params":{"year":"2012"}}],"join":[{"dataset":"MXZHQGZSVH8484K", "dataset_col":"city","joinset":"AXUEUK5UD2LTT9B", "joinset_col":"city"}]}

Response:

{"status":200,"message":"Success","body":[{"_id":"I6J1BOS3AOQHOVK4.3665285884961091369257549650","aggravated_assault":"644","arson2":"40","burglary":"2,951","city":"Fort Lauderdale","forcible_rape":"48","larcenytheft":"6,419","motor_vehicle_theft":"590","murder":"16","population1":"170,827","property_crime":"9,960","robbery":"835","state":"Florida","violent_crime":"1,543","year":"2012","poultry":"N","vegetables":"Y","updatetime":"5\/10\/12 17:59","seafood":"Y","schedule":"January - December Sunday 9:00 AM to 3:00 PM","trees":"Y","marketname":"Las Olas Outdoor Green Market","credit":"Y","y":"26.119545","x":"-80.131268","flowers":"N","soap":"Y","bakedgoods":"Y","eggs":"N","wine":"N","location":"Closed-off public street","sfmnp":"N","prepared":"Y","cheese":"N","street":"1201 E Las Olas Boulevard","plants":"Y","maple":"N","nuts":"Y","wic":"N","honey":"Y","jams":"Y","wiccash":"N","crafts":"Y","meat":"N","zip":"33301","snap":"Y","county":"Broward","fmid":"1002892","website":"","herbs":"Y","nursery":"Y"},{"_id":"I6J1BOS3AOQHOVK67.961077725234391369257704120","aggravated_assault":"497","arson2":"19","burglary":"841","city":"Oceanside","forcible_rape":"48","larcenytheft":"3,003","motor_vehicle_theft":"445","murder":"8","population1":"171,141","property_crime":"4,289","robbery":"175","state":"California","violent_crime":"728","year":"2012","poultry":"N","vegetables":"Y","updatetime":"5\/7\/12 13:02","seafood":"Y","schedule":"January - December Thursday 9:00 AM to 1:00 PM, (Closed Thanksgiving. Rain or shine market)","trees":"Y","marketname":"Oceanside Certified Farmers Market and Fair","credit":"Y","y":"33.196789","x":"-117.380033","flowers":"Y","soap":"Y","bakedgoods":"Y","eggs":"Y","wine":"N","location":"Closed-off public street","sfmnp":"N","prepared":"Y","cheese":"Y","street":"300 North Coast Highway","plants":"Y","maple":"N","nuts":"Y","wic":"Y","honey":"Y","jams":"Y","wiccash":"Y","crafts":"Y","meat":"N","zip":"92054","snap":"N","county":"San Diego","fmid":"1002393","website":"http:\/\/mainstreetoceanside.com\/Events\/FarmersMarket\/FarmersMarket.html","herbs":"Y","nursery":"Y"}]}

Error Messages

The API supresses normal HTTP status codes so that they do not disrupt the behavior of clients that cannot handle them elegantly. Instead it returns a status code in the response, along with a specific error messsage (if applicable).

500 Error Exversion cannot understand your request and cannot identify what the problem is. Usually this error is returned when invalid JSON has been posted to the API.

401 Access Denied Exversion understands the request but can't perform it because of the status of the user's account.

  • Not enough money in your account
  • You do not have permission to access this dataset
  • Missing or invalid API key
  • You are not authorized to do this. Bad access token

400 Bad Request Exversion can read your request, but cannot execute it.

  • This data is not available through API
  • Malformed search query
  • Only one join is allowed at this time
  • You must run a query on the first table
  • Dataset does not exists

200 Success Exversion understands your request and can execute it.