You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 3, 2020. It is now read-only.
Copy file name to clipboardExpand all lines: README.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ Tools for crawling data from metacritic.com (for educational purposes)
5
5
Is under your responsibility that you respect the [Terms of Use of Metacritic](https://www.cbsinteractive.com/legal/cbsi/terms-of-use#Acceptable_Use), especially the point 11.13
6
6
7
7
## Description
8
-
These tools are designed for creating a SQLite file with [different kind of data](https://github.com/MarkelFe/metacritic-crawler/blob/master/README.md#example) that extracts from [Metacritic](https://www.metacritic.com). You won't find the result of the crawl, like a database, as this data is protected by copyright apart from that the content varies very frequently. For more information of how it works [check out this.](https://github.com/MarkelFe/metacritic-crawler/blob/master/README.md#method)
8
+
These tools are designed for creating a SQLite file with [different kind of data](#example) that extracts from [Metacritic](https://www.metacritic.com). You won't find the result of the crawl, like a database, as this data is protected by copyright apart from that the content varies very frequently. For more information of how it works [check out this.](#method)
9
9
10
10
## Requisites
11
11
You can install all this packages with ```pip install -r requirements.txt``` or you can manually install them.
@@ -18,7 +18,7 @@ Scrapy has his own command line tool, you **shouldn't** use the default Python S
18
18
0. Be sure that you have Python 3 installed.
19
19
1.[Download the repository](https://github.com/MarkelFe/metacritic-crawler/releases) and travel to the repository folder through your OS Command Line Tool
20
20
2. Install all the requirements with ```pip install -r requirements.txt```
21
-
3. Run the following command ```scrapy runspider games.py -o gm.jl``` which will create a file called gm.jl. This file will include the links to all the Metacritic games, the process of completing this file will take around 40-80 minutes. You can [modify some parameters inline](https://github.com/MarkelFe/metacritic-crawler/blob/master/docs/flags.md)
21
+
3. Run the following command ```scrapy runspider games.py -o gm.jl``` which will create a file called gm.jl. This file will include the links to all the Metacritic games, the process of completing this file will take around 40-80 minutes. You can [modify some parameters inline](#modifiers)
22
22
4. Run the command ```scrapy runspider analyze.py``` which will create the database games.db. To complete this process, it will take around 2 hours.
23
23
5. Done! The file ```games.db``` includes all the information. Use your preferred SQLite reader.
24
24
@@ -48,4 +48,4 @@ This is an example of the result of running these scripts. The first line is the
48
48
## Meta
49
49
Markel F. – [@Markel_f](https://twitter.com/Markel_f)
50
50
Distributed under the BSD license. See ``LICENSE`` for more information.
0 commit comments