Schiphol-Code-Assignment documentation!

Contents:

Project info

• Output overview
  • Data trelliscopes
  • Snakemake pipeline overviews
• Project summary
  • Project goal
  • Motivation for not sticking to assignment requirements
    • Unoriginal results from previous internal results
    • Go beyond data analysis
    • My differentiating strength
  • Approach
  • What I focused on
    • File directory setup
    • Remote data source
    • Reproducible analysis pipeline
    • Sphinx documentation
  • What I left out
• Exploration summary
• Environment setup
  • Conda environments
  • Docker image
• Create Pandas profiling report
  • Script parameters
  • Read data
  • Create pandas-profiling reports
  • Save HTML output to file

Preprocessing scripts

• Create minimal model input
  • Parameters
  • Returns
    • File parameters
    • Imports
    • Utility functions
  • Check for duplicates by id
  • Filter out data from 2017
  • Output prediction target
  • Write output to CSV
• Merge base input and feature data
  • Parameters
  • Returns
  • File parameters
    • Libraries
  • Overview of the output data
• Create train/test split of the data
  • Parameters
  • Returns
    • Script parameters
    • Imports
  • Visualize train/test split over time
  • Overview of the output data

Feature scripts

• Time features from DateTime data
  • year, week_number, day_of_week, etc..
  • Parameters
  • Returns
    • Imports
    • Load data
  • Overview of the output data
• Features from destinations
  • Parameters
  • File parameters
    • Libraries
    • Read data
  • Destination features
  • Write output to CSV
  • Overview of the output data
• Rolling window features of delays
  • Parameters
    • Script parameters
    • Imports
  • Write output to CSV
• Get holiday data from RijksOverheid
  • NOTE THIS SCRIPT HAS BROKEN
    • Set parameters
    • Load packages
    • Utility functions
  • Main step to gather and write holiday data
  • Assert correct output
  • Information about processed holiday data

Model scripts

• Fit baseline model using flight delay averages
  • Parameters
  • Returns
    • Imports
  • Read data
  • Prediction model
    • Define model
    • Train model
    • Evaluate model
    • Calculate regression metrics
  • Plot some prediction results
  • Write output to output directory
  • Pickle output files for mlflow artifacts
  • Write output to CSV
  • Log to mlflow
  • Overview of the output data
• Fit CatBoost model using extended features
  • Parameters
  • Returns
  • Imports
  • Read data
    • Prediction model
  • Define model
  • Select features for catboost model
  • Perform random search for optimal parameters
  • Train model
  • Evaluate model
    • Plot some prediction results
  • Write output to output directory
    • Pickle output files for mlflow artifacts
    • Write output to CSV
  • Log to MLFlow
  • Overview of the output data

Indices and tables

• Index
• Module Index
• Search Page