tom/soundcloud-scrape
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Commit Graph

  • ba9be2ee3f Check if the user exists through their username element master Thomas Wade 2018-10-01 10:03:37 +09:30
  • 61df852e63 Safely get title string Thomas Wade 2018-09-30 16:02:33 +09:30
  • f8a6aa3980 Initialise counts to 0 Thomas Wade 2018-09-30 16:00:02 +09:30
  • 7316afb1a5 Add favourite tag, artist, and most produced tag to user nodes Thomas Wade 2018-09-30 15:07:26 +09:30
  • 1440e060e4 Show progress through user list Thomas Wade 2018-09-30 15:03:30 +09:30
  • 3588e5937f Add argument to skip scraping and just spit out a graph Thomas Wade 2018-09-30 15:02:45 +09:30
  • 65a5a5c63b Check for None before using Counter Thomas Wade 2018-09-30 14:59:45 +09:30
  • 99d77a574b Normalise all tags to lowercase Thomas Wade 2018-09-30 11:04:19 +09:30
  • f00be5782f Get favourite artist and tag for each user Thomas Wade 2018-09-30 11:03:48 +09:30
  • 93e7a01698 Add some spacing Thomas Wade 2018-09-30 10:29:35 +09:30
  • 2e378af361 Replace 35.6M with 1.6M in comment Thomas Wade 2018-09-29 15:25:13 +09:30
  • a2fdec3e27 Fixed 'On Tour' section breaking like count Thomas Wade 2018-09-29 15:24:45 +09:30
  • 3a2f5d1932 Add some comments Thomas Wade 2018-09-29 15:01:42 +09:30
  • 3c160b71c1 Added check for no likes or reposts on tracks Thomas Wade 2018-09-29 13:34:18 +09:30
  • 717d7ae6c0 Do not add peers to graph if they have not been processed Thomas Wade 2018-09-29 13:29:38 +09:30
  • bb88bbeb24 Convert human-readable numbers to ints Thomas Wade 2018-09-29 13:29:08 +09:30
  • 3e9a4b5f65 Fix like count shown as None Thomas Wade 2018-09-29 13:28:37 +09:30
  • 6ebaa47013 Fix issue where users' following lists would not be scraped Thomas Wade 2018-09-29 12:39:37 +09:30
  • 2cc2016d86 Properly implement persistence Thomas Wade 2018-09-27 19:23:12 +09:30
  • 16a9c5c638 Set default MAX_DEGREE to 2 Thomas Wade 2018-09-27 17:02:08 +09:30
  • babc4156b5 Add settings for max items per user Thomas Wade 2018-09-27 17:01:52 +09:30
  • f3d171a92b Flatten all track properties to strings Thomas Wade 2018-09-27 16:54:40 +09:30
  • 4e14ce2235 Check for null results and flatten to string Thomas Wade 2018-09-27 16:46:03 +09:30
  • 86d7958489 Use default flag for shelving Thomas Wade 2018-09-27 00:15:29 +09:30
  • 161d24e238 Use Shelve to store user and track dicts while scraping Thomas Wade 2018-09-26 14:37:50 +09:30
  • 82445f9d37 Ignore graph files Thomas Wade 2018-09-26 13:17:11 +09:30
  • 70f3a61040 Make graph from scraped data Thomas Wade 2018-09-25 20:43:54 +09:30
  • f0ba07e013 Merge find chain into single find Thomas Wade 2018-09-25 20:42:37 +09:30
  • 490a2ac22d Add dividers and some missing comments Thomas Wade 2018-09-25 19:44:52 +09:30
  • 829374fc6e Add some configuration options near start of script Thomas Wade 2018-09-25 19:37:55 +09:30
  • 295e3e3d12 Track users and tracks in a database-like fashion Thomas Wade 2018-09-25 19:09:05 +09:30
  • 6e5cfdfa0d url should be None by default Thomas Wade 2018-09-25 19:04:44 +09:30
  • 7de170efc9 Mark users as processed and check when printing Thomas Wade 2018-09-25 19:03:44 +09:30
  • 0c496ecd71 Add constructor for User class Thomas Wade 2018-09-25 19:01:07 +09:30
  • 9e9a56386f Organise tracks by url rather than artist-author mishmash Thomas Wade 2018-09-25 17:23:53 +09:30
  • fa9517a21d Remove track dict that was inside track class Thomas Wade 2018-09-25 17:22:55 +09:30
  • ae3e32cd21 Get track URLs Thomas Wade 2018-09-25 17:22:28 +09:30
  • 2e06dc231f Enforce limits Thomas Wade 2018-09-25 17:21:23 +09:30
  • 8ef89e373b Process follow(ers/ing) of each user Thomas Wade 2018-09-25 16:54:26 +09:30
  • fc1b2ca4cd WIP: adding differentiation between ego distance Thomas Wade 2018-09-24 13:11:21 +09:30
  • d6dbc472bd Scrape users' follower and following lists Thomas Wade 2018-09-24 12:57:33 +09:30
  • ca49b6d1b6 Get users' likes Thomas Wade 2018-09-20 01:00:15 +09:30
  • 539580de2a Ignore collections Thomas Wade 2018-09-20 00:59:39 +09:30
  • cdf78f0208 Sprinkle a bit of NoneType safety into the mix Thomas Wade 2018-09-20 00:59:13 +09:30
  • cde49030dc Use a list to store users' tracks and likes Thomas Wade 2018-09-20 00:57:33 +09:30
  • aff01870b5 Scroll until limit is reached Thomas Wade 2018-09-20 00:54:30 +09:30
  • 3decee9b03 List tracks Thomas Wade 2018-09-19 20:49:54 +09:30
  • 8b4c3f5489 Genericise Get_Tracks_Info to take any infiniscrolling list URL Thomas Wade 2018-09-19 20:48:21 +09:30
  • 33dd9edcdc Actually use element provided by loop Thomas Wade 2018-09-19 18:37:04 +09:30
  • 8c8631df7c Scroll through tracks page Thomas Wade 2018-09-19 15:11:22 +09:30
  • 5781fab7bd Add some better structure to status prints Thomas Wade 2018-09-19 09:49:01 +09:30
  • 3f8aa7304f Scrape tracks Thomas Wade 2018-09-18 20:01:38 +09:30
  • deed2cf88d Add specific package versions Thomas Wade 2018-09-18 18:58:23 +09:30
  • 30eebeef90 Extract basic info scraping into function Thomas Wade 2018-09-18 18:35:57 +09:30
  • e5b2e8861a Print what's happening while scraping Thomas Wade 2018-09-18 18:17:20 +09:30
  • 4f36369d84 Get follower and following count Thomas Wade 2018-09-18 18:13:07 +09:30
  • 51624e9715 Use user object where possible Thomas Wade 2018-09-18 18:08:13 +09:30
  • 9b9345887b Check if element exists before parsing Thomas Wade 2018-09-18 17:45:54 +09:30
  • b8a6a431bf Initialise to None Thomas Wade 2018-09-18 17:45:01 +09:30
  • 0846626460 Print out collected details of each user Thomas Wade 2018-09-18 17:39:42 +09:30
  • e94c4ac596 Add Bossfight to users_to_process Thomas Wade 2018-09-18 17:39:16 +09:30
  • 0eedc67144 Use User class instead of dictionary Thomas Wade 2018-09-18 17:38:48 +09:30
  • 7792e5cbda Correct copy-paste Thomas Wade 2018-09-18 16:35:43 +09:30
  • 3879d437e2 Store collected data in user_dict Thomas Wade 2018-09-18 15:54:37 +09:30
  • c45391c436 Move comment for consistency Thomas Wade 2018-09-17 23:58:49 +09:30
  • 7cb2e2a2ff Get username, track count, and like count Thomas Wade 2018-09-17 23:55:53 +09:30
  • 7c7c016e27 Disable image loading Thomas Wade 2018-09-17 23:55:18 +09:30
  • 65c92026d9 Add initialisation messages Thomas Wade 2018-09-17 23:55:01 +09:30
  • 8241702fb1 Don't get page in setup phase Thomas Wade 2018-09-17 23:54:06 +09:30
  • 6ca88dfd94 Capitalise ChromeDriver Thomas Wade 2018-09-17 23:52:21 +09:30
  • 8faf429e0c Initial commit Thomas Wade 2018-09-17 19:02:00 +09:30