Deploy a collection of Youtube playlists as a static site

Problem

We had collection of close to 2000 videos (still growing) organized in a specific structure of folders and sub-folders. We needed to make them available for students and people interested in this content in a way that is searchable, easy to navigate and also free from distraction.

Solution

We tried the following and went through several iterations:

• AWS S3 for hosting the videos - easy to synch the local folder structure with remote but financially not sustainable for the project
• Youtube for hosting the videos and navigating - cannot customize the navigation and search
• Youtube for hosting and Next.js for navigation/search - good initial solution that worked as alpha launch
• Youtube for hosting and Docusaurus 2 for navigation/search - launched as beta version as the built-in features of Docusaurus reduced the need for several custom components that would otherwise need to be built

Techologies used

• Next.js
• Docusaurus 2, a modern static website generator - beta version (current)
• Algolia, a fast index-based site search - beta version (current)
• Vercel, for hosting the static site - alpha and beta versions
• Youtube, for hosting the videos and using the API to get the playlist content
• Node, for writing the scripts to synch the videos and generate the static content from the video structure

How-to guide

You can clone the repository and follow this guide to deploy the project with your own content.

To contribute

Please follow this guide

Project structure

• The organization of the videos and link with the associated playlists in Youtube is contained in the content/tafsir.json file. Currently, this file is updated manually. Depending on the size of your collection and it's structure, you could create this file manually. For this project, it was created by writing a script that parsed the folder where the videos are organized, including the children folders, and generated the json file
• Whenever you add new videos to existing playlists or add new playlists, you need to update the tafsir.json file and run the scripts/synch.js. This will create or update json files for each playlist in the content/lessons folder. Please make sure that you create a .env file in your local folder that contains the following variable YOUTUBE_API_KEY that you can create by following this guide
• Generating the site content with scripts/generatemdx.js:
  • The videos folder contains the mdx files and folders that power the doc plug-in of docusaurus. The content of this folder is generated by running the scripts/generatemdx.js script
  • This script also creates a content/search.json file that is used for the algolia search index. Currently, the file is uploaded manually to the algolia dashboard. In future, the script should update the algolia index directly using the algolia API
  • This script also creates content/videocount.json that is used on the home page cards to display the number of videos in each section
• The src folder contains the pages folder that powers your index file and a components folder for any custom components that you build for the index page or any mdx pages
• To customize the settings of Docusaurus, use the docusaurus.config.js. This is explained in detail here

Build and deployment process

1. If you added new video/s to existing YT playlists, run scripts/synch.js to update the json files in content/lessons
2. If you added new playlist/s, update the content/tafsir.json with the playlistid key in the appropriate section and then run scripts/synch.js
3. Run scripts/generatemdx.js to re-create the mdx files
4. Upload and replace the json data for the algolia index with content/search.json
5. Commit the changes to your branch and push. Ideally, you would want to push to the dev branch first and test and then merge to the main branch to deploy to production (Follow this guide for deploying to Vercel)