The Enterprise File Ingestion Platform is a local-first scaffold for uploading, extracting, parsing, and normalizing files. It supports various file types and archives, extracts entities, and indexes records into OpenSearch.
From the README
Enterprise File Ingestion Platform What this project is Enterprise File Ingestion Platform is a local-first scaffold for: • uploading files (presigned direct-to-object-store PUT, multi-GB capable) • recursively extracting archives (ZIP, RAR, TAR, TAR.GZ/TGZ, 7Z — including nested and password-protected) • parsing content into structured records (TXT/LOG, CSV, JSON/JSONL, generic text) • extracting entities/IOCs from every record (IPv4, emails, URLs, domains, timestamps, MD5/SHA1/SHA256, CVE IDs, Bitcoin addresses) • normalizing infostealer log dumps into unified credential records (secrets hashed, never stored in plaintext) • indexing parsed records into OpenSearch • monitoring jobs and...
Skills
- Go
- Python
- TypeScript
- Shell
- CSS
