Cool, EDGAR is an amazing public service. I think they use Akamai as their CDN so the downloads are remarkably fast.
A few years ago I wrote an SGML parser for the full SEC PDS specification (super tedious). But I have trouble leveraging my own efforts for independent research because I don't have a reliable securities master to link against. I can't take a historical CUSIP from 13F filings and associate it to a historical ticker/return. Or my returns are wrong because of data errors so I can't fit a factor model to run an event study using Form 4 data.
I think what's missing is a serious open source effort to integrate/cleanse the various cheapo data vendors into something reasonably approximating the quality you get out of a CRSP/Compustat.
Securities master to link against - Interesting. Here's a pipeline off the top of my head
1. Get CUSIP, nameOfIssuer, titleOfClass using the Institutional Holdings database
2. Use the company metadata crosswalk to link CUSIP + titleOfClass to nameOfIssuer to get cik
https://github.com/john-friedman/datamule-data/blob/master/d...
(recompiled daily using GH actions)
3. Get e.g. us-gaap:EarningsPerShareBasic from the XBRL database. Link using cik. Types of stock might be a member - so e.g. Class A, Class B? Not sure there.
For form 4, not sure what you mean by event study. Would love to know!
1. Two AWS ec2 t4g.nano instances polling the SEC's RSS and EFTS endpoints. (RSS is faster, EFTS is complete).
2. When new submissions are detected, they are sent to the Websocket (t4g.micro websocket, using Go for greater concurrency).
3. Websocket sends signal to consumers.
Archive:
1. One t4g.micro instance. Receives notifications from websocket, then gets submissions SGML from the SEC.
2. If submission is over size threshold, compresses with zstandard.
3. Uploads submissions to Cloudflare R2 bucket. (Zero egress fee, just class A / B operations).
4. Cloudflare R2 bucket is proxied behind my domain, with caching.
RDS
1. ECS Fargate instances set to run daily at 9 AM UTC.
2. Downloads data from archive, then parses them, and uploads them into AWS dbt.medium MySQL RDS.
3. Also handles reconciliation for the archive in case any filings were missed.
Cool, EDGAR is an amazing public service. I think they use Akamai as their CDN so the downloads are remarkably fast.
A few years ago I wrote an SGML parser for the full SEC PDS specification (super tedious). But I have trouble leveraging my own efforts for independent research because I don't have a reliable securities master to link against. I can't take a historical CUSIP from 13F filings and associate it to a historical ticker/return. Or my returns are wrong because of data errors so I can't fit a factor model to run an event study using Form 4 data.
I think what's missing is a serious open source effort to integrate/cleanse the various cheapo data vendors into something reasonably approximating the quality you get out of a CRSP/Compustat.
Yep! Pretty sure it is still Akamai. Via testing I've noticed they cap downloads at ~6mbps from e.g. home internet, but not GitHub or AWS.
SGML parsing is fun! - I've opensourced a sgml parser here https://github.com/john-friedman/secsgml
Securities master to link against - Interesting. Here's a pipeline off the top of my head 1. Get CUSIP, nameOfIssuer, titleOfClass using the Institutional Holdings database 2. Use the company metadata crosswalk to link CUSIP + titleOfClass to nameOfIssuer to get cik https://github.com/john-friedman/datamule-data/blob/master/d... (recompiled daily using GH actions) 3. Get e.g. us-gaap:EarningsPerShareBasic from the XBRL database. Link using cik. Types of stock might be a member - so e.g. Class A, Class B? Not sure there.
For form 4, not sure what you mean by event study. Would love to know!
How it works:
Websocket:
1. Two AWS ec2 t4g.nano instances polling the SEC's RSS and EFTS endpoints. (RSS is faster, EFTS is complete). 2. When new submissions are detected, they are sent to the Websocket (t4g.micro websocket, using Go for greater concurrency). 3. Websocket sends signal to consumers.
Archive:
1. One t4g.micro instance. Receives notifications from websocket, then gets submissions SGML from the SEC. 2. If submission is over size threshold, compresses with zstandard. 3. Uploads submissions to Cloudflare R2 bucket. (Zero egress fee, just class A / B operations). 4. Cloudflare R2 bucket is proxied behind my domain, with caching.
RDS
1. ECS Fargate instances set to run daily at 9 AM UTC. 2. Downloads data from archive, then parses them, and uploads them into AWS dbt.medium MySQL RDS. 3. Also handles reconciliation for the archive in case any filings were missed.