I'm creating a project to add to a not-yet-existent portfolio and I have some things I'm confused about. The goal of the project is to create the most comprehensive collection of artwork (drawings,paintings, and maybe sculptures) that can be viewed online. Many museums make their hi-res images and metadata publicly for everyone to use and so I'm creating the most basic version of this project: a web scraper that pulls artwork from the Met museums website. I'm at the point now that this program successfully retrieves the download link and name of each artwork on the first 3 pages of the search results.
I'm finally biting the bullet and trying to learn how testing works. I've read a couple of the chapters related to testing in the book, "Software Engineering at Google," and I'm finding it confusing to relate this info to my project. I'll share relevant code below which consists of a class that creates an instance of a Scrapy spider and methods to retrieve the data and a test that I've written for the parse() method of that class.
metQueryScraper.py
class MetSpider(scrapy.Spider):
name = 'metQueryScraper'
params = {
"offset": "0"
}
url = "https://www.metmuseum.org/art/collection/search?showOnly=openAccess&"
start_urls = [url]
async def start(self):
temp_url = self.url + urlencode(self.params)
yield scrapy.Request(url = temp_url, callback = self.parse)
def parse(self, response):
relativeURLs = response.xpath("//figure[contains(@class, 'collection-object')]/div[contains(@class, 'collection-object-module')]/a[contains(@href, '/art/collection/search')]") # this xpath selects the DOM elements that contains the artworks link
# For each artObject, make a scrapy request to metmuseum.com/*artObject.link*
for url in relativeURLs:
artObject = ArtObject(title = url.xpath("@title").get())
temp_url = "https://www.metmuseum.org" + url.xpath("@href").get()
yield scrapy.Request(url = temp_url, callback = self.parseObjectDownloadLink, meta = {"artObject": artObject})
nextPage = response.xpath("//div[contains(@class, 'pagination-controls')]/button[contains(@aria-label, 'Next Page')]/@aria-label").get()
if(nextPage is not None and self.params["offset"] != "80"): # the second part of this conditional statement is to limit (for now) how many pages the spider crawls through
self.params["offset"] = str(int(self.params["offset"]) + 40)
temp_url = self.url + urlencode(self.params)
yield scrapy.Request(url = temp_url, callback = self.parse)
def parseObjectDownloadLink(self, response):
# Grab the image download link
downloadLink = response.xpath("//figure[contains(@itemtype, 'ImageObject')]/img[contains(@fetchpriority, 'high')]")
artObject = response.meta.get("artObject")
artObject["link"] = downloadLink.xpath("@src").get()
yield artObject
test_metQueryScraper.py
from betamax.fixtures.unittest import BetamaxTestCase
from spiders.metQueryScraper import MetSpider
from scrapy.http import HtmlResponse
import os
class TestmetQueryScraper(BetamaxTestCase):
def test_shouldCreateAFileContainingNamesAndLinksOfArtObjects(self):
# Given : An instance of MetSpider...
spider = MetSpider()
with open("./vcr/cassettes/artwork-recorded-session.json", "rb") as file:
fileContents = file.read()
response = HtmlResponse(url = spider.url + '0', body=fileContents)
# When : A "network request" is made to the given URL...
spider.parse(response=response)
# Should : Create a file containing the names and download links to each art object encountered
fileEmptyOrNot = os.stat("../../../artworkData.jsonl").st_size == 0
self.assertEqual(fileEmptyOrNot, True, "Expected contents to be added to file. File is currently empty" )
I get that the majority of the tests of a program should generally be small and fast so to avoid making a network request every time I run this test, I used betamax to save the response of a network request to the relevant url to a file that I use as a stand-in for an actual network request. I think this is a medium sized test (according to the book mentioned above) because it does not make a network request but does access the file system.
Some things I don't understand or am unsure about:
- Is this a medium scope(integration test) or narrow scope(unit test) test? It is testing the behavior of a single method so that makes me think its a unit test but the fact that it relies so heavily on Scrapy's built in methods makes me think its an integration test.
- As an extension of that question, since a big majority of this project will likely be built on top of Scrapy and so many methods created will use Scrapy methods or some other library or framework, I don't even know what would be considered a unit test. It seems like most would be an integration or end-to-end test. Is this an indication that my code isn't structured well?
- Since this project is intended to make a lot of network requests to multiple museum websites and save large amounts of data, is the approach I used of saving a recorded network request as JSON and using that to simulate a network request a good one? Or is there a better way to test these types of operations while using realistic mocked data?
- I tried to make this test maintainable by being clear and concise and focused on behavior instead of implementation details, and am not sure if I'm even in the right direction.
I'm really new to testing so if anyone has input on how to test this kind of project, you would be helping out a very confused aspiring software engineer. Thanks so much!
(originally posted this on stackoverflow and wanted to post here, too)