Project: ML In Production

Group Project

For this project, you will work in small teams to develop a production machine learning project. In this project you will be playing the role of a data science and data engineering team working at ‘citibike’, which provides bike sharing services in NYC. An interesting aspect of ‘citibike’ is that it is a hybrid between a business which earns revenue, and a part of the public transport infrastructure of ‘NYC’. Owing to its hybrid nature, citibike provides a wealth of public data which can be used to solve data science problems relating both to business and to municipal governance.

Your projects will be developed and deployed on a class-specific ‘databricks’/‘AWS’ workspace which has been logging the status of citibike stations every 5 minutes and station information each day using the citibike ‘GBFS’ feed. You will also have access to other data, including trip data and weather data accessible through ‘bigquery’, and any data that you think you might need for your project.

The timeline for the project is as follows:

  1. Team Formation (Weeks 1-4, by September 27th). This will take place on the course slack page. Ideal teams will be 3 or 4 students. You should message your classmates and post what problems you are interested in working on. If at the end of Week 4 you are unable to find a team, you must contact me and I will assign you to a team. It is important to be thinking about your project idea and working towards your project proposal towards the end of this phase. After you form your team, I will send you an invitation to a GitHub Classroom and DataBricks workspace.

  2. Project Proposal (October 4th, 10%). You will develop and submit a project proposal with a maximum length of 3 pages (not including references). The proposal should define the problem and outline your proposed solution. It should also answer as best as possible the following questions:

    1. What type of Machine Learning problem will you solve, and how might citibike, New York City, or New Yorkers use the outputs? You do not necessarily have to achieve these goals but they are useful to have in mind.

    2. What data sources are you planning to use, and which if any external data sources will you add? Each project is required to add at least one additional data set/data source and make it available as a documented table in the shared catalog for other teams to potentially use.

    3. What initial approach/model types will you use to solve the problem?

    4. How will you evaluate model performance?

    5. Will model training be online (model updates whenever new data is ingested) or batch (at scheduled intervals)?

    6. Do you anticipate any large computational needs? If you think you will need to use large neural network models, LLMs, GPUs/TPUs, make sure to let me know here.

    7. The key midterm check for this project is the development of a minimally viable product, which will demonstrate feasibility and which you will iteratively improve. What does your minimally viable product look like?

  3. Midterm Check In (November 2nd-7th, 20%) You will present/demonstrate a minimal viable product version (MVP) of your system to your professor. Your presentation should be about 10 minutes long. The first part should be a slideshow presentation (target 5 minutes and 3 slides) which pitches your project, gives a diagram of your proposed solution, and outlines what challenges remain. The second part is a 5 minute demo of your MVP. It is ok if your MVP is very basic, ideally I want to see that you have given good thought to your application, tried to make your MVP an end-to-end product, and have a good understanding of the challenges that await. Afterwards there will be a question and answer session about your project.

  4. Production Deployment and Final Presentation (November 16th to December 14th, 50%) Projects should be deployed in production starting on November 16th and their performance monitored for 4 weeks. This production period will culminate in a final presentation which will be scheduled with your professor during the week of the 14th. The presentation will include an introduction to your system, a live-demo, and a question and answer with the instructor about your project and codebase. Your final presentation should include measures of model performance from the production period.

  5. Project Writeup/Documentation (December 18th, 20%): The documentation for this project shall consist of four things:

  1. A short model card which describes the model, its intended purpose, training data, and performance. Follow the format of ‘skops’ or read the paper that introduced the concept here Model Cards for Model Reporting. Standardized model documentation is common across a variety of industries including banking, medicine, and artificial intelligence (where the model card is typical).
  2. A datasheet which documents the dataset(s) that you collected to support your project. This should describe the motivation, composition, collection process, and intended uses. See the guide here Datasheets for Datasets. Think of this as a data dictionary but with additional information on the provenance and purpose of the dataset.
  3. A short runbook which describes the mechanics of your production system, i.e. what jobs run and when, where fitted models, logs, and performance measures are stored, etc.
  4. A whitepaper/technical blog post. Target length of 2000 words. This aims to be less formal than an academic research paper. You should have sections of the problem definition, system design, machine learning components, application demonstration (ideally you can integrate parts of your application into your report), and a conclusion/reflection. A great example of a whitepaper (though not an app) is here offgrid ai. Some great examples of reports are here, here, and here.