Hello all, I am adding people to the github repository for our project, and also creating branches for each team. If people could post their github under their teams (and thereby confirm which team they are) in the Citibike-Project-Organization, I will make sure you get added to the project and The structure of our work is as follows:
In order to maintain our budget, we will use databricks free tier for interactive runs, model development, etc. This will prevent an accidental programming mistake from depleting our classwide credit budget. Other safeguards will be in place in order to prevent that. The primary work-space will only run code directly from the main branch of our github project, and only in a “jobs” format. Thus for production I will have to approve your merges/PRs from your team branch.
For development and demonstrating that your jobs/code will be safe, you will use databricks free edition workspaces. I’ve setup the code in the main repo so that you can replicate most of the workflow in free edition, though there will be some caveats. Free edition gives allows you experiment/mess around while remaining on the safe side. I’ll make a video gives a tour of both the databricks workspace and the free edition that I use for development so that you can see what is required. Signup for databricks free edition here: https://docs.databricks.com/gcp/en/getting-started/free-edition . I had to use a different email address for free edition than my main account. Not sure if that is a me problem or something you will need to do to.
Because we are sharing resources, be just a little extra careful with what you run, especially verify and test code that has been modified or generated with help of an LLM. It is easy to generate code that grows beyond your capacity to understand what it is doing, so put in extra effort to make such code easy to understand, as I’ll also need to understand and approve it. I’ll show you some basics on how to estimate the costs from free edition.
Looking forward to reading your project proposals, and seeing what we all create this semester,
Sincerely, George