Commits · main · OpenWebSearchEU-Public / Preprocessing Pipeline · GitLab

Snippets Groups Projects

Jun 12, 2025
- Update schema description of ID field · ff6e5c6d
  Sebastian Heineking authored 1 week ago
  
  ff6e5c6d
May 08, 2025
- Resilipipe v0.2.1: Increased memory efficiency of curlielabels module; Test... · 7aa3e2eb
  Sebastian Heineking authored 1 month ago
  7aa3e2eb
Mar 10, 2025
- Resilipipe v0.2.0: Added minimal HTML version of the main content of a page · 18cf7a55
  Sebastian Heineking authored 3 months ago
  
  18cf7a55
Jan 31, 2025
- Version 0.1.2 of Resilipipe: Added opengraph column, Anchor texts for... · 60ff670b
  Sebastian Heineking authored 4 months ago
  60ff670b
Nov 25, 2024
- Expanded the README on how to develop new modules. Correction to path in spark test · 2612ef1f
  Sebastian Heineking authored 6 months ago
  
  2612ef1f
Sep 18, 2024
- Minor updates to README and pyproject.toml · d4ae57ac
  Sebastian Heineking authored 9 months ago
  
  d4ae57ac
- Removed deprecated files · dcf55881
  Sebastian Heineking authored 9 months ago
  
  dcf55881
- Updated Resilipipe to be more independent of deployment specifics like S3... · 16bbf914
  Sebastian Heineking authored 9 months ago
  16bbf914
Jul 03, 2024
- Corrected empty output for content_type_other · 306632b4
  Sebastian Heineking authored 11 months ago
  
  306632b4
- Version 0.1.0 of Resilipipe · 95fd8179
  Sebastian Heineking authored 11 months ago
  
  95fd8179
Feb 23, 2024
- Updated paths in prepare script and secondary spark jobs · 3d962db4
  Sebastian Heineking authored 1 year ago
  
  3d962db4
Feb 21, 2024
- Added Dockerfile · f85f1497
  Sebastian Heineking authored 1 year ago
  
  f85f1497
- Updated project directory · f26987e2
  Sebastian Heineking authored 1 year ago
  
  f26987e2
- Updated README · 1ec6c0af
  Sebastian Heineking authored 1 year ago
  
  1ec6c0af
Feb 14, 2024
- Updated public version of the Preprocessing Pipeline · ed3ea5e8
  Sebastian Heineking authored 1 year ago
  
  ed3ea5e8
Sep 04, 2023
- Added a files folder to distribute larger files to cluster nodes · 46bb35cc
  Sebastian Heineking authored 1 year ago
  
  46bb35cc
Aug 29, 2023
- Added domain labeling to preprocessing based on Curlie.org · 0f265e69
  Sebastian Heineking authored 1 year ago
  
  0f265e69
Aug 28, 2023
- Updated pipeline with (1) Datacenter parameters in conf.yaml (2) Improved... · 8a158ff3
  Sebastian Heineking authored 1 year ago
  8a158ff3
Aug 07, 2023
- Initial commit · b73334e4
  Sebastian Heineking authored 1 year ago
  
  b73334e4