Common Crawl
In this phase, the focus is to store Common Crawl Index data, ranging from November 2020 (CC-MAIN-2020-50) to March 2018 (CC-MAIN-2018-13), as listed in https://index.commoncrawl.org.
Each month's Index File List (parsed from cc-index.paths.gz) is downloaded, packed into tarballs and stored on the filecoin network.
To retrieve a particular month's index files, click the corresponding month and metadata for retrieval will be shown.
Month of Crawl
| CC-MAIN-2021-10 | February 2021 Index |
| CC-MAIN-2021-04 | January 2021 Index |
| CC-MAIN-2020-50 | November 2020 Index |
| CC-MAIN-2020-45 | October 2020 Index |
| CC-MAIN-2020-40 | September 2020 Index |
| CC-MAIN-2020-34 | August 2020 Index |
| CC-MAIN-2020-29 | July 2020 Index |
| CC-MAIN-2020-24 | May 2020 Index |
| CC-MAIN-2020-16 | March 2020 Index |
| CC-MAIN-2020-10 | February 2020 Index |
| CC-MAIN-2020-05 | January 2020 Index |
| CC-MAIN-2019-51 | December 2019 Index |
| CC-MAIN-2019-47 | November 2019 Index |
| CC-MAIN-2019-43 | October 2019 Index |
| CC-MAIN-2019-39 | September 2019 Index |
| CC-MAIN-2019-35 | August 2019 Index |
| CC-MAIN-2019-30 | July 2019 Index |
| CC-MAIN-2019-26 | June 2019 Index |
| CC-MAIN-2019-22 | May 2019 Index |
| CC-MAIN-2019-18 | April 2019 Index |
| CC-MAIN-2019-13 | March 2019 Index |
| CC-MAIN-2019-09 | February 2019 Index |
| CC-MAIN-2019-04 | January 2019 Index |
| CC-MAIN-2018-51 | December 2018 Index |
| CC-MAIN-2018-47 | November 2018 Index |
| CC-MAIN-2018-43 | October 2018 Index |
| CC-MAIN-2018-39 | September 2018 Index |
| CC-MAIN-2018-34 | August 2018 Index |
| CC-MAIN-2018-30 | July 2018 Index |
| CC-MAIN-2018-26 | June 2018 Index |
| CC-MAIN-2018-22 | May 2018 Index |
| CC-MAIN-2018-17 | April 2018 Index |
| CC-MAIN-2018-13 | March 2018 Index |
Please select a month from left.
| Deal ID | Miner ID | Payload CID | Filename | Deal size in bytes | Date | Curated dataset |
|---|