Puppeteer is a JavaScript / Nodejs library that instruments Google Chrome or Chromium browers using the Chrome DevTools Protocol. Think of it as a way to write JavaScript to control every aspect of a Chrome browser. I wrote an article Getting Started with Puppeteer that walked through the process of boostrapping a Nodejs / Puppeteer app and turning a webpage into a queryable API using Puppeteer.

I originally became interested in Puppeteer because I had some inefficiences in my everyday life. One of those inefficiences was how tedious and monotonous grocery shopping is. Puppeteer was instrumental in this endeavor.

For what it’s worth, grocery shopping isn’t THAT bad. It can be kinda crazy when young kids are tagging along. GroceryJS has the additional advantage of being an interesting technical challenge, so here we are.

Breaking Down Grocery Shopping

Shopping for groceries is a procedural activity:

  • You look through your fridge and pantry for things you need and make a list
  • You walk, bike, or drive to the store
  • You walk the aisles adding things to your cart as you see them
  • Finally you pay and bring your groceries home

This process tends to happen every week or two, for large grocery orders.

Translating this to a computer program, the primary actions are:

  • Read items from a list
  • Search and add items to your cart
  • Prepare the cart for checkout & checkout

With this in mind, I built GroceryJS.

Breaking Down the Script

GroceryJS is broken up into several pieces of code after plenty of experimentation:

Source

The source is the data backend, it’s where the grocery list is held. It also is the holding place for the results of a grocery store run. For, GroceryJS, I started with a text (YAML) file, then transitioned to a Google Sheet. I found the Google Sheet to be something that is accessible from everywhere, desktop & mobile, without needing a bunch of UI. Google provides a pretty robust set of Nodejs libraries you can use to interact with the Google Drive and Sheet APIs.

My grocery list is stored in the first 2 columns and first 50 rows (arbitrary) in the first sheet.

The sheets-grocery-source.js has two primary actions after initializing a few objects.

The first gets the grocery list, getGroceryList().

async getGroceryList() {
    let spreadsheetId = this._spreadsheetId;
    let sheetsService = this._sheetsService;

    return await new Promise((resolve, reject) => {
        sheetsService.spreadsheets.values.get({
                spreadsheetId: spreadsheetId,
                range: 'A1:C50'
            }, (err, result) => {
            if (err) {
                reject(err);
            } else if (result && result.data && result.data.values) {
                let items = [];

                for (let i = 1; i < result.data.values.length; i++) {
                    let value = result.data.values[i];
                    items.push({ name: value[0], quantity: value[1] });
                }
                resolve(items);
            } else {
                resolve([]);
            }
        });
    });
}

The second adds the results of a particular shopping run to the Sheet, addShoppingResults().

async addShoppingResults(title, sheetId, results) {
        let sheetsService = this._sheetsService;
        let spreadsheetId = this._spreadsheetId;

        return new Promise((resolve, reject) => {
            let requests = [];
            let idx = 1;

            // convert results to an array we can write
            let data = [];

            let headers = [
                { userEnteredValue: { stringValue: 'Requested' } },
                { userEnteredValue: { stringValue: 'Item' } },
                { userEnteredValue: { stringValue: 'Price' } },
            ];
            data.push({ values: headers });

            for (let i = 0; i < results.length; i++) {
                let result = results[i];
                let row = [];
                row.push({ userEnteredValue: { stringValue: result.requested } });

                if (result.result) {
                    row.push({ userEnteredValue: { stringValue: result.result.title } });
                    row.push({ userEnteredValue: { numberValue: result.result.price } });
                }
                data.push({ values: row });
            }

            // add the sheet
            requests.push({ 
                addSheet: {
                    /* removed for brevity's sake */
                }
            });

            // updateCells request
            requests.push({ 
                /* removed for brevity's sake */
            });
            
            // auto size things
            requests.push({
                /* removed for brevity's sake */
            });

            // execute the batch update
            sheetsService.spreadsheets.batchUpdate({
                spreadsheetId: spreadsheetId,
                resource: { requests: requests }
            }, (err, result) => {
                    
                if (err) {
                    reject(err);
                } else {
                    resolve(`https://docs.google.com/spreadsheets/d/${spreadsheetId}/edit#gid=${sheetId}`);
                }
            });
        });
    }

Shopper

The Shopper contains all the code and actions that make up a successful trip to the grocery store. It’s built on top of utility library I wrote called puppet-helper.js.

Puppet Helper

The Puppet Helper contains all of things needed to interact with a modern web app, like clicking a button, given a CSS selector:

async clickButton(selector, clickCount = 1) {
    this.assertPageOpen();
    let button = await this._page.$(selector);

    if (button) {
        await button.click({ clickCount: clickCount });
    } else {
        throw new Error(`Unable to click ${selector}`);
    }
}

Or getting text from an HTML element:

async getTextFromElement(element) {
    this.assertPageOpen();
    return await this._page.evaluate(el => el.innerText, element);
}

As you can imagine, you can string enough of these actions together to mimic a user, shopping for groceries online.

The Lowes Shopper

More and more grocery stores offer online shopping services on the internet, allowing customers the convenience of shopping from their computer, tablet, or mobile phone. We shop at Lowes Foods, a North Carolina-based grocery store chain. Lowes Foods offers an online shopping service, Lowes Foods To Go. For $49-$99 annually (or $4 to $5 per order), you can order your groceries using their web app. Once you place your order, a Lowes Foods employee will shop your order and call you when they are finished (or if they had any questions). When the order is complete, you can pick it up or have it delivered.

Dad/mom-hack, shop for groceries online if you can help it. Taking young kids to the grocery store get’s wild! 😅

I spent a lot of time examining the front-end code for Lowes Foods To Go. I have determined that it is an Angular based progressive web app. It lends itself very well to automating with Puppeteer. Using puppet-helper.js, I can string together a few methods to get several things done.

Searching for groceries
async searchImpl(query) {
    this._logger.info(`Searching for ${query}`);
    let productDivs = null;
    await this._puppetHelper.clearText('#search-nav-input');
    await this._puppetHelper.enterText('#search-nav-input', query);
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#search-nav-search');
    await this._puppetHelper.wait(MID);
    // body > div:nth-child(5) > div > div > div.content-wrapper > div > lazy-load > ol
    let resultsDiv = await this._puppetHelper.getElement('ol.cell-container');

    if (resultsDiv) {
        productDivs = await this._puppetHelper.getElementsFromParent(resultsDiv, '.cell.product-cell');
    }
    return productDivs;
}
Logging in
async login(email, password) {
    this._logger.info(`Logging into account ${email}...`);
    await this._puppetHelper.goToUrl(SHOPPING_URL);
    await this._puppetHelper.clickButton('#loyalty-onboarding-dismiss');
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#shopping-selector-parent-process-modal-close-click');
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#nav-register');
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.enterText('#login-email', email);
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.enterText('#login-password', password);
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.clickButton('#login-submit');
    await this._puppetHelper.wait(XLONG);
}
Showing and emptying your cart
async showCart() {
    this._logger.info(`Opening the shopping cart...`);
    await this._puppetHelper.clickButton('#nav-cart-main-checkout-cart');
    await this._puppetHelper.wait(MID);
}

async emptyCart() {
    this._logger.info(`Emptying cart...`);
    await this.showCart();
    await this._puppetHelper.clickButton('#checkout-cart-empty');
    await this._puppetHelper.wait(NANO);
    await this._puppetHelper.clickButton('#error-modal-ok-button');
    await this._puppetHelper.wait(MINI);
}

Putting it Together

With all the pieces mentioned, I can have GroceryJS, prepare a shopping cart full of groceries. When it’s done, it sends me an email with a link to the cart (so that I can quickly checkout) and a link to the Google Sheet for tracking purposes.

(async () => {
    let shopper = null;

    try {
        let sheetSource = new SheetGrocerySource(logger, credential.client_email, credential.private_key, config.source.sheetId);
        await sheetSource.init();
        let list = await sheetSource.getGroceryList();

        // login and create a blank slate to shop
        shopper = new LowesShopper(logger);
        await shopper.init(config.shopper.headless);
        await shopper.login(config.shopper.email, config.shopper.password);
        await shopper.emptyCart();

        // do the shoppping
        let shoppingResults = [];
        for (let i = 0; i < list.length; i++) {
            let requestedItem = list[i];
            let shoppedItem = await shopper.addItemToCart(requestedItem.name, requestedItem.quantity);
            shoppingResults.push({ requested: requestedItem.name, result: shoppedItem });
        }

        // notify 
        let dateStr = moment().format('MMMM Do YYYY @ h:mm a');
        let title = `Shopping Trip on ${dateStr}`;

        let urlToCart = 'https://shop.lowesfoods.com/checkout/cart';
        let urlToSheet = await sheetSource.addShoppingResults(title, moment().unix(), shoppingResults);
        let emailBody = `
            <span><b>Shopping Cart:</b> ${urlToCart}</span><br />
            <span><b>Shopping Results:</b> ${urlToSheet}</span>`;
        let mailOptions = {
            service: config.email.sender.service,
            user: config.email.sender.email,
            password: config.email.sender.appPassword
        };
        mailUtil.sendEmail(config.email.recipeint.sender,
            config.email.recipeint.email,
            title, emailBody, mailOptions);
    } catch (e) {
        logger.error('Error while shopping', e);
    } finally {

        if (shopper) {
            await shopper.shutdown();
        }
    }
})();

Conclusion

So that’s it. GroceryJS isn’t done yet. The real work is actually in the details, like the algorithm for adding groceries from the search results to your cart. Lowes Foods To Go has it’s own search algorithm for determining the relevance of a result to a search. In many cases their algorithm will not match expectations, but it can be augmented:

  • Should GroceryJS prefer groceries that are on sale?
  • Should GroceryJS prefer groceries for a specific brand?
  • Should GroceryJS prefer groceries I’ve purchased before?

There are a ton of calculations we make every time we grocery shop that I didn’t realize until I started working on GroceryJS.

Hit up the GitHub repository for all the source code. Be sure to read the README file before jumping in.

github.com/emuneee/groceryjs

I’m really interested to see how many people fork this project and write GroceryJS applications for their preferred grocery stores.

Thanks for reading! I’m launching an email newsletter @ PuppetHero.com! If you like articles, tutorials, and news about Puppeteer, I encourage you to sign up. It’s completely free!

🧇