eccc_to_commons.git
18 months agoRewrite almanach merge logic master
Pierre Choffet [Tue, 25 May 2021 22:12:31 +0000 (18:12 -0400)]
Rewrite almanach merge logic

Partial input/merge files use was flawed: missing days were triggering errors.
Some traps were added to make sure we won't generate files with unexpected corner cases.

18 months agoFix almanach merger with empty pop values
Pierre Choffet [Tue, 25 May 2021 05:53:16 +0000 (01:53 -0400)]
Fix almanach merger with empty pop values

18 months agoFix almanach merger with empty normalMean temperatures
Pierre Choffet [Tue, 25 May 2021 05:28:07 +0000 (01:28 -0400)]
Fix almanach merger with empty normalMean temperatures

18 months agoDo not echo commands
Pierre Choffet [Tue, 25 May 2021 05:19:58 +0000 (01:19 -0400)]
Do not echo commands

18 months agoFix command in README
Pierre Choffet [Tue, 18 May 2021 18:56:11 +0000 (14:56 -0400)]
Fix command in README

18 months agoAdd merge Bash script
Pierre Choffet [Wed, 12 May 2021 20:23:12 +0000 (16:23 -0400)]
Add merge Bash script

18 months agoAdd ECCC almanach XML merger
Pierre Choffet [Wed, 12 May 2021 15:36:16 +0000 (11:36 -0400)]
Add ECCC almanach XML merger

2 years agoRequest bot flag at upload time
Pierre Choffet [Sun, 31 May 2020 03:36:37 +0000 (23:36 -0400)]
Request bot flag at upload time

2 years agoAdd beginning offset
Pierre Choffet [Fri, 29 May 2020 15:22:35 +0000 (11:22 -0400)]
Add beginning offset

2 years agoLess naive rate limiter
Pierre Choffet [Wed, 27 May 2020 17:46:50 +0000 (13:46 -0400)]
Less naive rate limiter

2 years agoReduce upload rate
Pierre Choffet [Sun, 19 Apr 2020 00:30:13 +0000 (20:30 -0400)]
Reduce upload rate

2 years agoAdd 1s wait after successful upload to prevent flooding server
Pierre Choffet [Tue, 24 Mar 2020 18:09:21 +0000 (14:09 -0400)]
Add 1s wait after successful upload to prevent flooding server

2 years agoRemove categories as patch sadly didn't make it for initial import
Pierre Choffet [Tue, 24 Mar 2020 14:46:44 +0000 (10:46 -0400)]
Remove categories as patch sadly didn't make it for initial import

2 years agoFix documentation
Pierre Choffet [Mon, 9 Mar 2020 15:53:08 +0000 (11:53 -0400)]
Fix documentation

2 years agoPrevent empty files to be generated
Pierre Choffet [Fri, 6 Mar 2020 17:56:14 +0000 (12:56 -0500)]
Prevent empty files to be generated

ECCC sometimes provides almanac files containing no data at all. We won't convert them into JSON file anymore.

2 years agoRestrict uploaded file pattern
Pierre Choffet [Fri, 6 Mar 2020 04:48:59 +0000 (23:48 -0500)]
Restrict uploaded file pattern

2 years agoLicence change
Pierre Choffet [Thu, 5 Mar 2020 22:01:39 +0000 (17:01 -0500)]
Licence change

While in theory weather data is in canadian public domain due to they are no more than facts, ECCC has chosen to distribute them in OGL. Publishing as CC0 - especially for international distribution - would be illegal due to it has no attribution requirements. However, OGL is known to be compatible with CC BY 4.0 so have to stick with it for now.

2 years agoFix wikicode field name
Pierre Choffet [Wed, 4 Mar 2020 23:26:52 +0000 (18:26 -0500)]
Fix wikicode field name

2 years agoAdd script to perform the upload
Pierre Choffet [Thu, 6 Feb 2020 20:49:57 +0000 (15:49 -0500)]
Add script to perform the upload

It's a generic Mediawiki batch upload script that takes files in a directory (recursively) and uploads them in a Mediawiki while keeping their tree.

Also add login_password into .gitignore to actively prevent contributors to shoot themselves in the foot.

2 years agoAdd missing almanac to commons stylesheet
Pierre Choffet [Thu, 6 Feb 2020 20:25:09 +0000 (15:25 -0500)]
Add missing almanac to commons stylesheet

2 years agoSwitch from station id to climate id
Pierre Choffet [Tue, 4 Feb 2020 22:39:39 +0000 (17:39 -0500)]
Switch from station id to climate id

2 years agoAdd required scripts to generate almanac JSONs
Pierre Choffet [Fri, 24 Jan 2020 20:38:29 +0000 (15:38 -0500)]
Add required scripts to generate almanac JSONs

2 years agoFix indentation
Pierre Choffet [Fri, 24 Jan 2020 20:13:28 +0000 (15:13 -0500)]
Fix indentation

2 years agoAdd missing dependency
Pierre Choffet [Thu, 23 Jan 2020 22:17:56 +0000 (17:17 -0500)]
Add missing dependency

2 years agoComplete process from ECCC website to Commons files generation
Pierre Choffet [Wed, 22 Jan 2020 18:49:27 +0000 (13:49 -0500)]
Complete process from ECCC website to Commons files generation

2 years agoAdd transformation to fix ECCC XML files
Pierre Choffet [Wed, 23 Oct 2019 23:52:16 +0000 (19:52 -0400)]
Add transformation to fix ECCC XML files

The monthly files are provided with an error on the root element. The eccc_fixer.xslt fixes it so it can be validated before processing.