Thomas Wade
3c160b71c1
Added check for no likes or reposts on tracks
2018-09-29 13:34:18 +09:30
Thomas Wade
717d7ae6c0
Do not add peers to graph if they have not been processed
2018-09-29 13:29:38 +09:30
Thomas Wade
bb88bbeb24
Convert human-readable numbers to ints
2018-09-29 13:29:08 +09:30
Thomas Wade
3e9a4b5f65
Fix like count shown as None
2018-09-29 13:28:37 +09:30
Thomas Wade
6ebaa47013
Fix issue where users' following lists would not be scraped
...
Wrong URL ^^'
Also 'followed' was replaced with 'following' everywhere for clarity and consistency with the changes
2018-09-29 12:39:37 +09:30
Thomas Wade
2cc2016d86
Properly implement persistence
...
This will help immensely with finding that /one/ bug that seems to occur several hours into scraping
2018-09-27 19:23:12 +09:30
Thomas Wade
16a9c5c638
Set default MAX_DEGREE to 2
2018-09-27 17:02:08 +09:30
Thomas Wade
babc4156b5
Add settings for max items per user
2018-09-27 17:01:52 +09:30
Thomas Wade
f3d171a92b
Flatten all track properties to strings
2018-09-27 16:54:40 +09:30
Thomas Wade
4e14ce2235
Check for null results and flatten to string
2018-09-27 16:46:03 +09:30
Thomas Wade
86d7958489
Use default flag for shelving
...
Avoids an issue when running the script for the first time where Shelve would freak out and die because the shelf doesn't exist.
2018-09-27 00:15:29 +09:30
Thomas Wade
161d24e238
Use Shelve to store user and track dicts while scraping
...
Just in case something breaks mid-scrape
2018-09-26 14:40:26 +09:30
Thomas Wade
82445f9d37
Ignore graph files
2018-09-26 13:17:11 +09:30
Thomas Wade
70f3a61040
Make graph from scraped data
2018-09-25 20:43:54 +09:30
Thomas Wade
f0ba07e013
Merge find chain into single find
2018-09-25 20:42:37 +09:30
Thomas Wade
490a2ac22d
Add dividers and some missing comments
2018-09-25 19:44:52 +09:30
Thomas Wade
829374fc6e
Add some configuration options near start of script
2018-09-25 19:37:55 +09:30
Thomas Wade
295e3e3d12
Track users and tracks in a database-like fashion
...
Rather than have several copies of a user/track object, collect them all in a single dictionary and give the keys to each user object. This will avoid duplication and make generating graphs easier.
2018-09-25 19:09:05 +09:30
Thomas Wade
6e5cfdfa0d
url should be None by default
2018-09-25 19:04:44 +09:30
Thomas Wade
7de170efc9
Mark users as processed and check when printing
2018-09-25 19:03:44 +09:30
Thomas Wade
0c496ecd71
Add constructor for User class
...
This was causing issues where appending to any of the lists would update the class variable rather than the instance variable, leading to everyone having the same likes, tracks, etc.
2018-09-25 19:01:07 +09:30
Thomas Wade
9e9a56386f
Organise tracks by url rather than artist-author mishmash
2018-09-25 17:23:53 +09:30
Thomas Wade
fa9517a21d
Remove track dict that was inside track class
...
wtf? why was this here?
2018-09-25 17:22:55 +09:30
Thomas Wade
ae3e32cd21
Get track URLs
2018-09-25 17:22:28 +09:30
Thomas Wade
2e06dc231f
Enforce limits
2018-09-25 17:21:23 +09:30
Thomas Wade
8ef89e373b
Process follow(ers/ing) of each user
2018-09-25 16:55:43 +09:30
Thomas Wade
fc1b2ca4cd
WIP: adding differentiation between ego distance
2018-09-24 13:11:21 +09:30
Thomas Wade
d6dbc472bd
Scrape users' follower and following lists
2018-09-24 12:58:01 +09:30
Thomas Wade
ca49b6d1b6
Get users' likes
2018-09-20 01:00:15 +09:30
Thomas Wade
539580de2a
Ignore collections
...
I really don't want to iterate over collections so we'll just ignore them outright
2018-09-20 00:59:39 +09:30
Thomas Wade
cdf78f0208
Sprinkle a bit of NoneType safety into the mix
2018-09-20 00:59:13 +09:30
Thomas Wade
cde49030dc
Use a list to store users' tracks and likes
2018-09-20 00:58:48 +09:30
Thomas Wade
aff01870b5
Scroll until limit is reached
2018-09-20 00:54:30 +09:30
Thomas Wade
3decee9b03
List tracks
2018-09-19 20:49:54 +09:30
Thomas Wade
8b4c3f5489
Genericise Get_Tracks_Info to take any infiniscrolling list URL
2018-09-19 20:48:21 +09:30
Thomas Wade
33dd9edcdc
Actually use element provided by loop
...
*facedesk*
2018-09-19 18:37:04 +09:30
Thomas Wade
8c8631df7c
Scroll through tracks page
2018-09-19 15:11:22 +09:30
Thomas Wade
5781fab7bd
Add some better structure to status prints
2018-09-19 09:49:01 +09:30
Thomas Wade
3f8aa7304f
Scrape tracks
...
Some scrolling is still needed to actually get the whole track list
2018-09-18 20:01:38 +09:30
Thomas Wade
deed2cf88d
Add specific package versions
2018-09-18 18:58:23 +09:30
Thomas Wade
30eebeef90
Extract basic info scraping into function
2018-09-18 18:35:57 +09:30
Thomas Wade
e5b2e8861a
Print what's happening while scraping
2018-09-18 18:21:30 +09:30
Thomas Wade
4f36369d84
Get follower and following count
2018-09-18 18:13:07 +09:30
Thomas Wade
51624e9715
Use user object where possible
2018-09-18 18:08:13 +09:30
Thomas Wade
9b9345887b
Check if element exists before parsing
2018-09-18 17:45:54 +09:30
Thomas Wade
b8a6a431bf
Initialise to None
2018-09-18 17:45:01 +09:30
Thomas Wade
0846626460
Print out collected details of each user
2018-09-18 17:39:42 +09:30
Thomas Wade
e94c4ac596
Add Bossfight to users_to_process
2018-09-18 17:39:16 +09:30
Thomas Wade
0eedc67144
Use User class instead of dictionary
2018-09-18 17:38:48 +09:30
Thomas Wade
7792e5cbda
Correct copy-paste
2018-09-18 16:35:43 +09:30
