Increase the GitHub rate limit (generic)¶
1 - Introduction ¶
Back to TOC¶
GitHub enforces a rate limit of 60 API calls per hour. for unauthenticated users. When working with Text-Fabric datasets that consist of a large number of files (such as the CenterBLC/N1904 dataset, has at least 60 files), this rate limit can be easily reached during the loading process, effectively blocking loading of the full dataset by any unauthenticated user.
There are two possible solutions to this problem:
During the creation of our dataset we produced a zip file containing the complete dataset and attached it to the release published on GitHub. Whenever a user downloads the latest release, the Text-Fabric code downloads this zip file without using the risk of hitting the rate limit. Hence, most users will not hit this rate limit when using the latest dataset, provided this includes a complete.zip file. See also the Text-Fabric documentation on how this is done by the dataprovider using A.zipall().
If you however needs to load another (previous) version, there is a need to raise the rate limit, which can be done by adding authentication, after which the rate limit is raised to 5000 API calls per hour.
The procedure in this NoteBook provides guidance on how to authenticate as user on GitHub by installing a personal access token.
2 - The High-Level procedure¶
Back to TOC¶
The process basicly consist of the following two high level steps:
After performing these steps, the Text-Fabric code finds this personal token and will add it to every GitHub API call it make which will allow for the increased rate limit.
As this personal token functions as your personal credential, do not pass it on to others, let them obtain their own!
2.1 - Create a personal access token¶
This step consist of the following actions:
2.1.1 - Login on GitHub¶
In order to obtain a personal access token it is required to login using your personal login on GitHub.
Go to https://github.com/login. This will bring up the following login screen:
Login using your credentials or create a new account.
2.1.2 - On GitHub create the personal access token¶
After login on GitHub create the personal token (classic) using this procedure. The image below provides a screendump of the procedure as per August 2024:
The last step (copy the token) is important. Store the token localy as you need it for the next part of the procedure.
2.2 - Install the personal token in your environment¶
After you have obtained your personal token en and put it in an environment variable named GHPERS on the system where your app runs.
The method used to install this personal access token depends on your operating system. A good starting point is to consult the official Text-Fabric documentation
See here if you want to set the variabale on windows using the command prompt.