聯系方式

您當前位置:首頁 >> Web作業Web作業

日期:2019-03-15 10:36

Resources / Assignments (/COMP9321/19T1/resources/22490)

/ Assignment 2 (/COMP9321/19T1/resources/24962)

/ Data Service for World Bank Economic Indicators

Data Service for World Bank Economic

Indicators

Data Service for World Bank Economic

Indicators

Change Log: Add some tips about the submission, and fix the spec inconsistent problem.

Assignment template : https://www.cse.unsw.edu.au/~cs9321/19T1/assn/assn2.tar.gz

(https://www.cse.unsw.edu.au/~cs9321/19T1/assn/assn2.tar.gz)

You shall develop it with Flask (1.0.2), Flask-SQLAlchemy (2.3.2) and flask-restplus (0.12.1)[optional] .

As for you are safe to install the three packages with pip3 on option "--user". You are only allowed to

install those three packages if one of them not available.

You can use any package already installed in CSE machine.

In this assignment, you are asked to develop a Flask-Restplus data service that allows a client to read

some publicly available economic indicator data for countries around the world, and allow the consumers

to access the data through a REST API.

IMPORTANT : For this assignment , you will be asked to access the given Web content programmatically.

Some Web hosts do not allow their web pages to be accessed programmatically, some hosts may block

your IP if you access their pages too many times. During the implementation, download a few test pages

and access the content locally - try not to perform too many programmatically.

The source URL: http://api.worldbank.org/v2/ (http://api.worldbank.org/v2/)

(http://api.worldbank.org/v2/) Documentations on API Call Structure:

(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)

https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure

(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)

(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)

The World Bank indicators API provides 50 year of data over more than 1500 indicators for countries

around the world.

List of indicators can be found at: http://api.w o rldbank.org/v2/indicators

(http://api.worldbank.org/v2/indicators)

List of countries can be found at: (http://api.worldbank.org/v2/countries)

http://api.worldbank.org/v2/countries (http://api.worldbank.org/v2/countries)

As the documentation shows , you can construct URLs specifying different parameters to acquire

necessary data. Use a custom URL to get information about a specific indicator. For example, the data on

the GDP of all countries from 2013 to 2018 can be acquired via this URL:

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017)

http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2013:2018&format=json

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json)

For this assignment we are interested in annual values of a user specified indicator from 2013 to 2018 for

one or all countries. The content of the file is quite straightforward. The data service specification will ask

you to 'import' this data into a 'data storage'. You should inspect the content of the file carefully and

decide on a data model and storage method. You will find that a JSON format is suitable to accessing data

and publishing it.

The database we used in this assignment is SQLite . You need to determine the way about how to store

the data. We don't care about your output format as long as your data is correct and reasonable

arranged. Please download the python code template a2.py. Don't change the function signature

"create_db" which you need to use it to create the database. All the other function, it's flexible. Also,

no auto-test will be provided.

Note: You may need some additional key in the record for storage purpose.

Don't send too much requests to the source API as it will ban your IP address, be careful.

Please make sure your have a swaggerUI page on you API.

How to create database in SQLite: http://www.sqlitetutorial.net/sqlite-python/creating-database/

(http://www.sqlitetutorial.net/sqlite-python/creating-database/)

Assignment Specification

The data service should use JSON format to publish its data and implement following operations.

1- Import a collection from the data service

This operation can be considered as an on-demand 'import' operation. The service will download the

JSON data for all countries (http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2012:2017&format=json) respective to the year 2013 to 2018 and identified by the indicator id given

by the user and process the content into an internal data format.

Parameters should be given to the endpoint (in the payload) by the user:

indicator_id : an indicator (http://api.worldbank.org/v2/indicators)

http://api.worldbank.org/v2/indicators (http://api.worldbank.org/v2/indicators)

After importing the collection, the service should return a response containing at least the following

information:

location : the URL with which the imported collection can be retrieved

collection_id : a unique identifier automatically generated

Specification Make Submission Check Submission Collect Submission

creation_time : the time the collection stored in the database

Example:

HTTP operation: POST /<collections>

Input Payload:

{ "indicator_id" : "NY.GDP.MKTP.CD" }

Returns: 201 Created

{

"location" : "/<collections>/<collection_id>",

"collection_id" : "<collection_id>",

"creation_time": "2019-03-09T12:06:11Z",

"indicator" : "<indicator>"

}

The return response contains the location of the data entry created as the result of the processing

the import.

You should return appropriate responses in case of invalid indicators or any invalid attempts to use

the endpoint ( e.g. If the input indicator id doesn't exist in the data source, return error 404 )

If an input contains a n indicator that already has been imported before, you should still return the

location of the data entry - but with status code 200 OK (instead of 20 1 Created).

A POINT TO PONDER ABOUT: An `asynchronous POST'?? If a POST takes too long, you may not

want the client to wait. What you would do? You do not need to address this in the assignment.

The source API has pagination; in order to get all of data you need to send many request to import a

single collection; however, you are required to get only first two pages instead of all:

http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2013:2018&format=json&page=2

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2012:2017&format=json&page=2)

The data entries inside the collection must be converted as described below:

Data entry conversion:

Here is an example of source data entry as it is in the source API

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2013:2018&format=json) :

{

"indicator": {

"id": "NY.GDP.MKTP.CD",

"value": "GDP (current US$)"

},

"country": {

"id": "1A",

"value": "Arab World"

},

"countryiso3code": "",

"date": "2016",

"value": 2513935702899.65,

"unit": "",

"obs_status": "",

"decimal": 0

}

However, you do not need to store all of its attributes; instead convert it to a JSON format as below:

{

"country": "Arab World",

"date": "2016",

"value": 2513935702899.65

}

And as a result a collection should be formatted and stored in the database as follow:

{

"collection_id" : "<collection_id>",

"indicator": "NY.GDP.MKTP.CD",

"indicator_value": "GDP (current US$)",

"creation_time" : "<creation_time>"

"entries" : [

{ "country": "Arab World", "date": "2016", "value": 2513935702899

.65 },

...

{ "country": "Caribbean small states", "date": "2017", "value":

68823642409.779 },

...

]

}

2- Deleting a collection with the data service

This operation deletes an existing collection from the database. The interface should look like as below:

HTTP operation: DELETE /<collections>/{collection_id}

Returns: 200 OK

{

"message" :"Collection = <collection_id> is removed from the database!"

}

3 - Retrieve the list of available collections

This operation retrieves all available collections. The interface should look like as like below:

HTTP operation: GET /<collections>

Returns: 200 OK

[

{

"location" : "/<collections>/<collection_id_1>",

"collection_id" : "collection_id_1",

"creation_time": "<time>",

"indicator" : "<indicator>"

},

{

"location" : "/<collections>/<collection_id_2>",

"collection_id" : "collection_id_2",

"creation_time": "<time>",

"indicator" : "<indicator>"

},

...

]

4 - Retrieve a collection

This operation retrieves a collection by its ID . The response of this operation will show the imported

content from world bank API for all 6 years. That is, the data model that you have designed is visible here

inside a JSON entry's content part.

The interface should look like as like below:

HTTP operation: GET /<collections>/{collection_id}

Returns: 200 OK

{

"collection_id" : "<collection_id>",

"indicator": "NY.GDP.MKTP.CD",

"indicator_value": "GDP (current US$)",

"creation_time" : "<creation_time>"

"entries" : [

{"country": "Arab World", "date": "2016", "value": 2513935702899

.65 },

{"country": "Caribbean small states", "date": "2017", "value": 6

8823642409.779 },

...

]

}

5 - Retrieve economic indicator value for given country and

a year

The interface should look like as like below:

HTTP operation: GET /<collections>/{collection_id}/{year}/{country}

Returns: 200 OK

{

"collection_id": <collection_id>,

"indicator" : "<indicator_id>",

"country": "<country>,

"year": "<year">,

"value": <indicator_value_for_the_country>

}

6 - Retrieve top/bottom economic indicator values for a

given year

The interface should look like as like below:

HTTP operation: GET /<collections>/{collection_id}/{year}?q=<query>

Returns: 200 OK

{

"indicator": "NY.GDP.MKTP.CD",

"indicator_value": "GDP (current US$)",

"entries" : [

{

"country": "Arab World",

"date": "2016",

"value": 2513935702899.65

},

...

]

}

The <query> is an optional parameter which can be either of following:

top<N> : Return top N countries sorted by indicator value

bottom<N> : Return bottom N countries sorted by indicator value

where N can be an integer value between 1 and 100. For example, a request like " GET /<collections>/<

id>/2015?q=top10 " should returns top 10 countries according to the collection_id.

Notes:

REASON : the source API will ban your (and tutors') IP addresses if you send too many requests

The source API has pagination; in order to get all of data you need to send many request to import a

single collection; however, you are required to get only first two pages instead of all:

http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2013:2018&format=json&page=1

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2012:2017&format=json&page=1)

http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2013:2018&format=json&page=2

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2012:2017&format=json&page=2) or to get all data use :

http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2013:2018&format=json&per_page=2000

(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?

date=2012:2017&format=json&per_page=2000)

In Q6: top1 should return the highest numeric value

Your code must implemented in flask-restplus and provide swagger doc for testing the endpoints.

Late Penalties:

2 marks per day.

Resource created 7 days ago (7 days ago), last modified about 23 hours ago (about 23 hours ago).

Submission:

When you make a submission, please make sure your create_db function can make a db for you so that

we can test it. DB's names doesn't matter, just make sure it's same as what you used on your API.

give cs9321 assn2 a2.py

Plagiarism:

this will result in a final 0 mark for COMP9321 2019 Term 1.

Comments

! " (/COMP9321/19T1/forums/search?forum_choice=resource/24864)

# (/COMP9321/19T1/forums/resource/24864)

$ Add a comment

Meng Sun (/users/z5149213) about 3 hours ago (Thu Mar 14 2019 06:31:53 GMT+1100 (AEDT))

Hi professor,

For the source data of the 2 page json, could my program save them locally as files. And

the read these files to operate? Or I have to visit the website every time when I need

them.

Reply

Xiaocong Chen (/users/z5027195) about 2 hours ago (Thu Mar 14 2019 06:51:12 GMT+1100

(AEDT))

When testing you can do that as it may ban your ip due to many requests. But for

submission version, you have to using the API version instead file

Reply

Meng Sun (/users/z5149213) about 2 hours ago (Thu Mar 14 2019 07:33:25 GMT+1100

(AEDT))

Thanks.

As for the db file, could we save it locally?

Reply

Elliott Vercoe (/users/z3411256) about 4 hours ago (Thu Mar 14 2019 05:15:17 GMT+1100 (AEDT))

For q6, if the query is malformed, do we throw a 400 error or do we just ignore the

query?

So if I sent GET /<collections>/< id>/2015?q= top6a4 should I send a 400, or just ignore

the query and send all entries for that year, as though this query wasnt there?

Reply

Xiaocong Chen (/users/z5027195) about 4 hours ago (Thu Mar 14 2019 05:31:25 GMT+1100

(AEDT))

If the thing you request is not exist just raise the error

Reply

Yingjie Zheng (/users/z5213329) about 8 hours ago (Thu Mar 14 2019 01:00:42 GMT+1100

(AEDT)), last modified about 8 hours ago (Thu Mar 14 2019 01:02:47 GMT+1100 (AEDT))

Hi.

It's hard to tell which part has been update. And it's a long article.

Can you provide some "diff" when you update the spec? Say, if you use git for this

article and share the repo, we can simply view the git history and easily notice any

change.

That will help.

Reply

Xiaocong Chen (/users/z5027195) about 4 hours ago (Thu Mar 14 2019 05:31:57 GMT+1100

(AEDT))

check below

Reply

Yingjie Zheng (/users/z5213329) about 3 hours ago (Thu Mar 14 2019 05:58:17

GMT+1100 (AEDT))

I don’t understand. What are you referring by “below”?

The question Meng Sun ask? That has nothing to do with my question.

Reply

Meng Sun (/users/z5149213) about 10 hours ago (Wed Mar 13 2019 23:10:13 GMT+1100 (AEDT))

Hi everyone, I have a question, how do we send the http operation? do we need to

design a UI for the user to input and make opration?

Reply

Elliott Vercoe (/users/z3411256) about 8 hours ago (Thu Mar 14 2019 01:24:32 GMT+1100

(AEDT))

you can use curl. if you're on windows, go to command prompt and type

curl -i -H "Content-Type: application/json" -X POST -d "{\"indicator_id\" :

\"NY.GDP.MKTP.CD\"}" http://localhost:5000/collections

Reply

Kaylen Payer (/users/z5076219) about 11 hours ago (Wed Mar 13 2019 22:17:25 GMT+1100

(AEDT)), last modified about 11 hours ago (Wed Mar 13 2019 22:17:57 GMT+1100 (AEDT))

Hi Everyone,

I've got a basic hello world flask app running fine on my local machine. However, on

when I ssh into cse i have issues:

When i run the app and use `curl http://0.0.0.0:4444/` i get:

"Failed to connect to 0.0.0.0 port 4444: Connection refused"

I'm using the following commands to start the flask app (since the default port was

giving me "flask OSError: [Errno 98] Address already in use")

```

a pp = Flask(__name__)

if __name__ == '__main__':

app.run(debug=True, host=os.getenv('IP', '0.0.0.0'), port=int(os.getenv('PORT',

4444)))

```

If anyone could suggest how i am able to test this on the uni server without physically

logging in to a physical machine, I would be very appreciative for any guidance.

Thank you :)

Reply

Xiaocong Chen (/users/z5027195) about 11 hours ago (Wed Mar 13 2019 22:43:52

GMT+1100 (AEDT))

don’t try to hold services at 0.0.0.0. Try 127.0.0.1

Reply

Kaylen Payer (/users/z5076219) about 10 hours ago (Wed Mar 13 2019 22:59:35

GMT+1100 (AEDT))

Thanks for getting back to me Xiaocong. I did originally try using 127.0.0.1, and

have just tested this again, but it still gives the same connection refused error :(

Reply

Xiaocong Chen (/users/z5027195) about 10 hours ago (Wed Mar 13 2019 23:10:50

GMT+1100 (AEDT))

hmm, you can go to one of the labs and ask for help.

Reply

Elliott Vercoe (/users/z3411256) about 24 hours ago (Wed Mar 13 2019 09:40:44 GMT+1100

(AEDT))

Has anyone had any luck with getting cascade deletes working on SQLAlchemy? I've

used similar packages, but sqlalchemy seems to hate deleting children when the parents

are deleted.

Reply

Michael Tran (/users/z3461919) about 23 hours ago (Wed Mar 13 2019 10:03:14 GMT+1100

(AEDT))

it’s a limitation with SQLite I believe. When you start a connection, you need to

redefine the relationship.

Something something pragma something.

Would give a more concrete explanation, but I’m sleep-typing

Reply

Chaoran Huang (/users/z5022460) about 3 hours ago (Thu Mar 14 2019 06:26:24

GMT+1100 (AEDT))

I removed your post on testing. As you know this is an individual assignment, and

sharing such testing can be an at least lv2 academic misconduct.

Reply

Michael Tran (/users/z3461919) about 3 hours ago (Thu Mar 14 2019 06:47:21

GMT+1100 (AEDT))

Understood.

Reply

Elliott Vercoe (/users/z3411256) about 8 hours ago (Thu Mar 14 2019 01:25:53 GMT+1100

(AEDT))

thanks Michael. I gave up on trying to set up ondelete cascade and am just

manually deleting the required instances. this is a lot faster than troubleshooting

it.

Reply

Yingjie Zheng (/users/z5213329) a day ago (Wed Mar 13 2019 09:03:58 GMT+1100 (AEDT)), last

modified about 24 hours ago (Wed Mar 13 2019 09:37:11 GMT+1100 (AEDT))

Dear teaching stuff:

I notice there might be an inconsistency between the following statements:

"After importing the collection, the service should return a response containing at least

the following information:" (location, collection_id, creation_time).

"You do not have to return the actual imported content with POST. Just return the

location . "

Please confirm it (and fix it, if applicable).

BTW: I also counter same question Christopher Chaaya

(https://webcms3.cse.unsw.edu.au/users/z5016114) asked 3 days ago . Would you please

take a look at it?

Reply

Xiaocong Chen (/users/z5027195) about 23 hours ago (Wed Mar 13 2019 10:07:10

GMT+1100 (AEDT))

Yeah, fixed, please check now

Reply

Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:17:26 GMT+1100 (AEDT))

For <collections> please refer to please read https://restful-apidesign.readthedocs.io/en/lates...

(https://restful-apidesign.readthedocs.io/en/latest/resources.html)

Reply

Christopher Chaaya (/users/z5016114) about 23 hours ago (Wed Mar 13 2019 10:31:18

GMT+1100 (AEDT))

So am I understanding this correctly - you want us to create a separate resource for

every inputted <collections> in the URL?

For example, say we fetch the indicator "NY.GDP.MKTP.CD" from the remote by

POSTing to an endpoint: "/hello" and it results in a collection with ID 1.

We can then retrieve this collection with a GET /hello/1. BUT, we cannot retrieve it

with a GET /byebye/1.

HOWEVER - If we then POST the same indicator "NY.GDP.MKTP.CD" to the endpoint

"/byebye" - we should now be able to access it using a GET /byebye/1?

Is this correct?

Reply

Load More Comments

Yipu Ding (/users/z5180553) a day ago (Wed Mar 13 2019 07:04:49 GMT+1100 (AEDT))

Sorry, I still don't get the meaning of <collections>. If it is an arbitrary string, then that

would mean we seperate different collections, but isn't that what the collection_id in the

layer below is doing?

So shouldn't <collections> just be some default string?

Reply

Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:16:20 GMT+1100 (AEDT))

please read https://restful-api-design.readthedocs.io/en/lates... (https://restful-apidesign.readthedocs.io/en/latest/resources.html)

Reply

Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:14:41 GMT+1100 (AEDT))

Collection means a collections of web resources here, the resources are essentially

returned required data.

Reply

Fengting Yang (/users/z5089358) a day ago (Wed Mar 13 2019 05:16:41 GMT+1100 (AEDT))

What does <collections> means? A collection name defined by user?

Reply

Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:16:30 GMT+1100 (AEDT))

please read https://restful-api-design.readthedocs.io/en/lates... (https://restful-apidesign.readthedocs.io/en/latest/resources.html)

Reply

Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:12:15 GMT+1100 (AEDT))

Collection means a collections of web resources here, the resources are essentially

returned required data

Reply

Fengting Yang (/users/z5089358) about 22 hours ago (Wed Mar 13 2019 10:57:34

GMT+1100 (AEDT))

can you please give some possible values on <collections>

Reply


版權所有:編程輔導網 2018 All Rights Reserved 聯系方式:QQ:99515681 電子信箱:[email protected]
免責聲明:本站部分內容從網絡整理而來,只供參考!如有版權問題可聯系本站刪除。

25选5一等奖多少钱