Python Tools for Data Collection

Overview

Python script tools are essential for automating data collection and processing, leveraging the vast libraries and frameworks available in Python. These tools effectively extract data from various sources, such as APIs and online databases, ensuring timely and accurate data collection, which is crucial for maintaining up-to-date datasets. By automating the extraction process, Python scripts minimize the risk of human error while saving considerable time and effort.

Once the data is collected, Python script tools can standardize, format, and clean the raw data, preparing it for upload to ECAstats. Python scripts streamline the data-handling process from collection to preparation, making them essential for data-driven projects.

Python Scripts Tools for Data Collection

• Sustainable Development Goal (SDG)
• Gross Domestic Product (GDP)
• Balance of Payment (BOP)
• Agriculture (FAOSTAT)
• Education
• Government Finance
• Human Development
• Labour
• Mo Ibrahim Governance
• UN Comtrade
• World Bank

Key Features

• Automated Data Retrieval: Extract data from various APIs and online databases with minimal manual intervention.
• Data Cleaning & Processing: Standardizes, formats, and cleans raw data to align with UNECA’s ECAstats.
• Multi-Format Support – Exports collected data to CSV and Excel for easy access.
• Flexible Data Export: Export processed data into CSV and Excel files in pivoted and non-pivoted formats.
• Data Upload: Prepares Excel files for uploading to ECAstats

USE

1. Sustainable Development Goals (SDG)

• Scripts: Extract SDG indicators from the United Nations SDG API national and regional levels from https://unstats.un.org/home/ UN Stats

2. Gross Domestic Product (GDP)

• Scripts: Extract GDP data from the United Nations GDP, API national and regional levels from https://unstats.un.org/home/ UN Stats

3. Balance of Payments (BOP)

• Scripts: Extract Balance of Payments data from the IMF for https://www.imf.org/en/Data

 4. Agriculture (FAOSTAT)

• Scripts: Gather production, trade, and food security data from FAOSTAT API: http://www.fao.org/faostat/en/

5. Education

• Scripts: Extract education statistics from UNESCO API: http://data.uis.unesco.org/

 6. Government Finance

• Scripts: Extract public finance and expenditure data from IMF API https://www.imf.org/en/Data

7. Human Development

• Scripts: Extract Human Development related indicators from UNDP API: http://hdr.undp.org/en/data

 8. Labor Market

• Scripts: Extract Labor Market from ILO and national labor bureaus https://www.ilo.org/data-and-statistics.

 9. Mo Ibrahim Governance

• Scripts: Extract Mo Ibrahim Foundation performance indicators from Mo Ibrahim Foundation: https://mo.ibrahim.foundation/

10. UN Comtrade

• Scripts: Extract international trade statistics for import/export UN Comtrade API: https://comtrade.un.org/

11. World Bank Data

• Scripts: Extract macroeconomic and social indicators from World Bank API: https://data.worldbank.org/

Resources

Python Script Repository

• https://github.com/mhunegnaw/UNECA_Data_Collection_Automation

How to Get Started

• Open Jupyter Notebook and navigate to your script.
• Open the script file in Jupyter Notebook.
• Run the script to collect and process data.
• The script will generate and save Excel files.
• Check the files for accuracy and completeness.
• Log in to ECAstats and upload the Excel files.

Contact

African Centre for Statistics

Email: ecastats@un.org